2014-01-31

apt-install

Do you ever get tired of typing
sudo apt-get update && apt-get install <package>
just to install one package that you added to your DEB repo? I do and I decided to do something about it. What I really miss is the intelligence of yum which simply updates its repo caches if they are too old.

apt-install (github.com/schlomo/apt-install) is the next best thing. It is a simply Python script that updates the cache and installs the packages given as command line arguments. And it shows a nice GUI with a progress bar:


Turns out that the parts are all there and part of aptdaemon. The only part missing was putting them together into this little script:
Please note that I actually completely don't understand how to write async code. I'll be happy about all feedback with better implementations.

2014-01-24

Simple Video Tricks

While working on the new Recorder (see also last posting) I suddenly faced several challanges with the resulting video files:
  • Many short chunks (50MB each, about 30-60 seconds) need to be merged
  • Extract the actual talk from a longer recording, e.g. the recorder was on for one hour but the talk was only half an hour
  • Convert the video into another container format because Adobe Premiere does not like AVI files
  • Create video thumbnails
  • Convert videos to be compatible with HTML5 <video> playback
Turns out that avconv (or ffmpeg) is the swiss army knife for all of these tasks! I am not qualified to say which is better, for my purposes the avconv that ships with Ubuntu is good enough. The examples given here work with both. When I write avconv I mean both tools.

Since I don't want to degrade the video quality through repeated decode/encode steps I always use -codec copy after the input file to simply copy over the audio and video data without reencoding it.

Concatenate Videos

This is probably the easiest part: avconv has an input protocol called concat which simply reads several files sequentially:

avconv -i concat:file1|file2 -codec copy out.mp4

Note: This method simply concatenates the input files as-is. It is mostly useful for transport streams. This methods usually fails to concatenate files that have container metadata. For that you can use the concat demuxer that is only available in ffmpeg. Full details can be found in the ffmpeg Wiki.

This script automates the process. It takes two files as arguments and concatenates all the files in the same directory starting from the first one and ending with the last one. Further arguments are passed to avconv.

Convert Video Container

One of the most common tasks is to convert the video container, e.g. from AVI to MOV or MP4:

avconv -i input.avi -codec copy output.mov

Create Video Thumbnails

Another simple task. Extract an image from the video after 4 seconds:

avconv -i input.avi -ss 4 -vframes 1 poster.jpg

Or for all the videos in a directory:

ls *.mp4 | while read ; do avconv -i "$REPLY" -vframes 1 -ss 4 -y "$REPLY.jpg" ; done


Create HTML5 Video

Convering a video for HTML5 playback requires keeping some standards, e.g. using a compatible profile and level. The following worked very well for me:

avconv -i INPUT-FILE -pix_fmt yuv420p -c:v libx264 -crf 30 -preset slower -profile:v Main -level 31 -s 568x320 -c:a aac -ar 22050 -ac 1 -b:a 64k -strict experimental out.mp4 ; qt-faststart out.mp4 OUTPUT-FILE.mp4 ; rm out.mp4

For smartphone consumption this is more than good enough. The result is about 3MB/min. You can increase the video size (e.g. 854x480) or quality (e.g. -crf 28), but this will also significantly increase the file size. The qt-faststart moves some MP4 metadata to the beginning of the file so that the file can be streamed.

Create video from image

Simple way to create a video of 3 seconds length at 25 frames/sec from a PNG image:

avconv -loop 1 -i image.png -r 25 -t 3 video.mp4


2014-01-15

Hostname-based Access Control for Dynamic IPs

Sometimes less is more. The most simple way to protect my private web space on my web server is this:

<Location />
    Order Deny,Allow
    Deny from All
    Allow from home.schapiro.org
</Location>

But what to do if home.schapiro.org changes the IP every 24 hours and if the reverse DNS entry (PTR) is something like p5DAE56B9.dip0.t-ipconnect.de? When my computer at home connects to the web server the source IP address is used for a reverse DNS lookup. This lookup returns the above mentioned provider-assigned name and not home.schapiro.org,  the web server will never be able to identify this IP as belonging to my home router.

The solution is to write the IP↔Name mapping for my dynamic IPs into /etc/hosts. That way a reverse lookup on the IP will actually yield the information from /etc/hosts and not ask the DNS system.

Since I don't want to do this manually every time my IP changes, I automate it with this script. It reads host names from /etc/hosts.autoupdate and injects them into /etc/hosts:

The script is actually part of the hosts-updater DEB package which also installs a man page and a CRON job to run this every 5 minutes. As a result my own server recognizes my dynamic IPs as authorized and under their "proper" name.

2014-01-08

Simple UDP Stream Recorder

At the office I got a 3 channel digital Audio/Video Recorder to conveniently record our talks without much human effort. The device has an analog video input for the video camera (standard resolution) and a digital video input (Full HD) and an audio input.
Epiphan VGADVI Recorder
These 3 inputs will be merged into a single side-by-side video where you can see the speaker next to his computer output. The video can be even larger than Full HD, for example 2688x1200 (a 768 pixels wide SD image next to a 1920 pixels wide HD image):

The device is far from cheap (list price is 1840 € + VAT) and can really do a lot. For example, it can create H.264 movies with a bitrate of up to 9 Mbit. It can also upload the videos to a CIFS share, but sadly that works only at a transfer speed of about 4 Mbit! So how could I transfer the videos at really high quality settings (9 Mbit) to the CIFS share? Waiting 2 hours to transfer the videos of a 1 hour talk is no option.


Linux and Open Source to the rescue!


My solution is a simple UDP stream recorder running on my desktop that receives a UDP stream of the video and saves it directly onto the CIFS share :-) UDP broadcasting of the video stream is a nice feature of the device and works at the full bitrate that it can encode. This is actually meant to be used for live broadcasts of the talk to the Internet, but it also can serve to beam the video from the device to another computer.

Since it took quite a while to cook up this simple solution (and I did not find any satisfactory search result), here it is:

Implementation Details

This implementation uses some tricks to make it so simple:
  • socat connects between incoming UDP packets on port 6002 and a program that writes the data to a file
  • Upstart keeps socat running and restarts it if it fails
  • socat terminates itself if it did not receive anything for 5 seconds
  • The Perl code
    • uses perl -n as the read loop
    • creates the destination file only if there is actually some data to write.
      No data = no file
    • uses the current date and time as filename - at the time it actually starts to receive something
  • Logging (only errors) is done via syslog
As a result each time the recorder is used the videos (H.264 in a MPEG TS container) will appear instantly on the file share. Just connect the device and the UDP Stream Recorder automatically records everything.