Feature #221

Need daemon to handle launch/shutdown of ffmpeg

Added by Hammel about 5 years ago. Updated about 5 years ago.

Status:ClosedStart date:30 Aug 2013
Priority:ImmediateDue date:
Assignee:Hammel% Done:

100%

Category:04 - Root File System
Target version:1.0 - Atreides
Severity:02 - High

Description

The PiBox media server web server needs to launch ffmpeg before it returns a page to the remote browser. Launching it is easy from php before the streaming starts, but we need to kill it when the browser leaves that page.

To do that we need javascript in the page served to perform an action. That action will cause the ffmpeg session associated with that page to stop.

The best way to handle this is to have a daemon that launches ffmpeg and associates a tag with it. PHP generates the tag and tells the daemon to launch ffmpeg. The tag is returned in the Javascript in the served page so that javascript can call a URL that is PHP that pulls the tag in the GET variables and tells the daemon to stop the ffmpeg session.

Associated revisions

Revision 4ad5b5ed
Added by Hammel about 5 years ago

RM #221: Stubbed support for queue processing. This is what will determine if a process needs to be started or needs to be stopped.

Revision a2f2955c
Added by Hammel about 5 years ago

RM #221: Added initial support for starting/stopping ffmpeg streams, but this is untested. Unit tests need updating.

Revision de88cf48
Added by Hammel about 5 years ago

RM #221: Extend unit tests for testing queue processor that starts streams, and fix minor issues found from those tests.

Revision c2b25be9
Added by Hammel about 5 years ago

RM #221: Fix null pointer bug by moving test for null a little eariler.

Revision 237331c5
Added by Hammel about 5 years ago

RM #221: Added test for multiple webcam streams. Added negative queue processor tests. Added timed test.

Revision e161f4a0
Added by Hammel about 5 years ago

RM #221: Implemented heartbeat processing. Added thread to expire abandonded streams. Updated unit tests to support both.

Revision 16fbde7e
Added by Hammel about 5 years ago

RM #221: Completed init script. Added packaging (opkg) targets.

Revision b7dd60d3
Added by Hammel about 5 years ago

RM #221: Added cross compile script.

Revision 4c29e2f3
Added by Hammel about 5 years ago

RM #221: Initialize "filename" before each use. Change loggerLocal to logger in case we add multiple destinations later.

Revision 57333d79
Added by Hammel about 5 years ago

RM #221. Don't free filename unless it's non-null. Change loggerLocal to logger in case we add multiple destinations later.

Revision 1d0a2baa
Added by Hammel about 5 years ago

RM #221: Reference message in logger before freeing message, not after. Change loggerLocal to logger in case we add multiple
destinations later.

Revision fa66b131
Added by Hammel about 5 years ago

RM #221: Null terminate execvp args. Change loggerLocal to logger in case we add multiple destinations later.

Revision efdd90e4
Added by Hammel about 5 years ago

RM #221: Change loggerLocal to logger in case we add multiple destinations later.

Revision 44a53412
Added by Hammel about 5 years ago

RM #221: Cleanups related to valgrind-reported issues based on short-term unit tests.

Revision 3190d3ff
Added by Hammel about 5 years ago

RM #221: Updated unit tests to make it easier to run individual subtests. Fixed leaks exposed in various subtests as reported by
valgrind. Timed tests of 300 seconds now reports no leaks.

History

#1 Updated by Hammel about 5 years ago

  • % Done changed from 0 to 10
The daemon should:
  1. Accept inbound XML messages
  2. Validate them
  3. Generate a UUID
  4. Return the UUID or NACK

The accepted XML is parsed and a data structure placed on a queue that is handled by a separate thread to manage child processes.

XML format is TBD. Using XML allows me to use a template file within the web server where tags can be replaced by PHP to specify what ffmpeg should be opening and how. libxml2 is already available in PiBox's rootfs.

I need a way to start an autoconf project. Some possible tutorials:

#2 Updated by Hammel about 5 years ago

  • % Done changed from 10 to 30

I've created a new project, piboxd, under the pibox umbrella on gitorious. An autoconf-based build has been created and I've started work on the core of the daemon. So far I've got command line parsing, logging, embedded build date/version, daemon mode and initial support for queueing of inbound messages. The server thread that accepts inbound messages also has simple nc-based unit tests.

I've got a preliminary design for the daemon including the simplistic protocol for messages to it. Messages are one-way, which means a request comes in and is processed by the daemon but is not acknowledged. This design has a block diagram in a Dia project which will be included in the source tree for piboxd.

After thinking about the original description of the daemon I've started to feel there might need to be one additional message from the browser (via javascript, probably through a PHP proxy request) that lets the daemon know the page is being viewed. If the daemon doesn't get this within a few seconds (it's very close to the browser, physically, in the trailer) then it should kill the process it just started. The javascript on the web page could repeat this heartbeat message periodically. This is a safeguard against abandoned streams. Additionally, the stream could have a time limit placed on it so it times out after a while. A timeout on the web cam would be useful. A timeout on streamed movies might not be.

#3 Updated by Hammel about 5 years ago

  • % Done changed from 30 to 40

Update on development of the daemon:

There is a small protocol for messaging between the web interface and the daemon. That protocol is structurally supported but not all features are implemented yet. Most importantly, support for starting and stopping a webcam stream is implemented. Unit tests are written for this functionality as well, though they don't verify that crtmpserver connects to the stream. They only test that launching ffmpeg using the correct command line is attempted.

What's next:
  1. Add test for multiple webcam stream requests. Only one request for the webcam can be active at at time. This is implemented in the daemon, but not tested yet.
  2. Add negative tests for queue processor. That would be tests that send badly formatted stream requests.
  3. Add duration test for queue processor. That would be a test that starts and stops tests over an extended period of time to test long term stability.
  4. Implement heartbeat processing. This should update the specified tag's heartbeat field with a new timestamp. That timestamp is local - when the message was received by the daemon.
  5. Implement thread that handles automatic shutdown of a stream if no heartbeat is received within a configurable timeout period. Be sure to call shutdown for thread from main().
  6. Write init script for daemon.
  7. Add packaging (opkg) for daemon.
  8. Verify cross compile works.

After that, I cam return to the web interface to write the PHP code that calls the daemon and then the Javascript that provides the heartbeat call and the termination call to the PHP.

#4 Updated by Hammel about 5 years ago

  • Added test for multiple webcam streams.
  • Added negative queue processor tests.
  • Added timed test.
  • Fixed minor bugs exposed by tests.
What's left:
  1. Implement heartbeat processing. This should update the specified tag's heartbeat field with a new timestamp. That timestamp is local - when the message was received by the daemon.
  2. Implement thread that handles automatic shutdown of a stream if no heartbeat is received within a configurable timeout period. Be sure to call shutdown for thread from main().
  3. Write init script for daemon.
  4. Add packaging (opkg) for daemon.
  5. Verify cross compile works.

I should also run the daemon with Valgrind and the timed duration test to see if I'm leaking anything.

#5 Updated by Hammel about 5 years ago

  • % Done changed from 40 to 50
  • Implemented heartbeat processing.
  • Added thread to expire abandonded streams.
  • Updated unit tests to support both.
What's left:
  • Write init script for daemon.
  • Add packaging (opkg) for daemon.
  • Verify cross compile works.
  • Run the daemon with Valgrind and the timed duration test to see if it leaks

#6 Updated by Hammel about 5 years ago

  • % Done changed from 50 to 60
  • Completed init script.
  • Added packaging (opkg) targets.
What's left:
  • Verify cross compile works.
  • Run the daemon with Valgrind and the timed duration test to see if it leaks

#7 Updated by Hammel about 5 years ago

  • % Done changed from 60 to 80
  • Added cross compile script.
What's left:
  • Run the daemon with Valgrind and the timed duration test to see if it leaks

#8 Updated by Hammel about 5 years ago

Running with valgrind found some problems. I'm fixing them piece by piece. There are a couple small leaks still left to identify, I think.

The command to run for basic valgrind coverage is:

    valgrind --tool=memcheck --leak-check=yes --show-reachable=yes --num-callers=20 --track-fds=yes src/piboxd -v3

This is taken verbatim (except for the program being tested) from http://cs.ecs.baylor.edu/~donahoo/tools/valgrind/. I'm running the following minimal test to search for leaks, currently:

    ./server.sh -v -t procs -t proce

#9 Updated by Hammel about 5 years ago

  • % Done changed from 80 to 90

I've cleaned up all valgrind reported leaks. One interesting one is that valgrind reports a dlopen "still reachable" leak if I use pthread_exit() when my threads exit. To avoid this I now call return(0) instead since the thread itself will be cleaned up immediately afterward. That's because where this happens is when the daemon is exiting anyway.

So now valgrid runs cleanly running simple tests. I'm going to run the timed test and make sure there are no long term leaks. Once those (if any) are cleaned up I'm done with this version of the daemon.

#10 Updated by Hammel about 5 years ago

Rats. There are some issues when the timed test is run. This may be due to either a long-run leak or there may be specific tests that generate these leaks. I need to run each unit test manually first, then the timed test to find if what's causing these. The output from valgrind follows.

==22581== 960 bytes in 12 blocks are indirectly lost in loss record 1 of 4
==22581==    at 0x4A074CD: malloc (vg_replace_malloc.c:236)
==22581==    by 0x402D99: queueMsg (msgQueue.c:352)
==22581==    by 0x402823: msgProcessor (msgProcessor.c:352)
==22581==    by 0x3474207D8F: start_thread (in /lib64/libpthread-2.14.90.so)
==22581== 
==22581== 1,035 bytes in 150 blocks are indirectly lost in loss record 2 of 4
==22581==    at 0x4A074CD: malloc (vg_replace_malloc.c:236)
==22581==    by 0x4026A1: msgProcessor (msgProcessor.c:312)
==22581==    by 0x3474207D8F: start_thread (in /lib64/libpthread-2.14.90.so)
==22581== 
==22581== 1,131 (1,040 direct, 91 indirect) bytes in 13 blocks are definitely lost in loss record 3 of 4
==22581==    at 0x4A074CD: malloc (vg_replace_malloc.c:236)
==22581==    by 0x402D99: queueMsg (msgQueue.c:352)
==22581==    by 0x402823: msgProcessor (msgProcessor.c:352)
==22581==    by 0x3474207D8F: start_thread (in /lib64/libpthread-2.14.90.so)
==22581== 
==22581== 13,104 (11,200 direct, 1,904 indirect) bytes in 140 blocks are definitely lost in loss record 4 of 4
==22581==    at 0x4A074CD: malloc (vg_replace_malloc.c:236)
==22581==    by 0x402E36: queueMsg (msgQueue.c:352)
==22581==    by 0x402823: msgProcessor (msgProcessor.c:352)
==22581==    by 0x3474207D8F: start_thread (in /lib64/libpthread-2.14.90.so)
==22581== 
==22581== LEAK SUMMARY:
==22581==    definitely lost: 12,240 bytes in 153 blocks
==22581==    indirectly lost: 1,995 bytes in 162 blocks
==22581==      possibly lost: 0 bytes in 0 blocks
==22581==    still reachable: 0 bytes in 0 blocks
==22581==         suppressed: 0 bytes in 0 blocks

#11 Updated by Hammel about 5 years ago

  • Status changed from In Progress to Closed
  • % Done changed from 90 to 100

Updated unit tests to make it easier to run individual subtests. Fixed leaks exposed in various subtests as reported by valgrind. Timed tests of 300 seconds now reports no leaks.

The daemon is completed, though its not tested on the hardware to verify it correctly starts the webcam stream. However, any bugs uncovered in that testing will be opened as new bugs.

This issue can be closed.

Also available in: Atom PDF