Truths or lies - decide yourself

On demand audio streaming with icecast

2011-10-22T21:58:00.002+02:00

The project goal was to make the impossible happen: Turn an icecast streaming server into an audio-on-demand server.

The background is, that I bought a NAS, which is basically a PC with Atom CPU. After erasing the firmware and installing Ubuntu Server on a 10 TB Raid 5 system, I was thinking what else I could do with the box.

Live streaming via icecast to my Wifi-radio worked for some time now, but this needs a running PC with a soundcard. What I had in my mind, was different:

It should run exclusively on the NAS, no need to switch on a PC
It should support an arbitrary number of playlists, each one corresponding to an icecast URL.
The upstream mechanism should work on demand because encoding many mp3 streams in parallel overloads the Atom CPU.
The current song should be shown in the display of the radio

Song titles
For the last requirement I added live metadata updating to the API for gmerlin broadcasting plugins. After learning, that Vorbis streams with changing song titles make my radio reboot, I wrote an MP3 broadcasting plugin (with libshout and lame). It seems that later firmware versions for the radio fix the vorbis problem, but the firmware update requires a windows software.

Commandline recorder
The recording and broadcast architecture for gmerlin was already working reliably, so I wrote a plugin, which takes a gmerlin album (=playlist), shuffles the tracks and makes them available as if it record from a soundcard. In addition, I wrote a commandline recorder, which could be started from a script. There is one script for starting a broadcast:

$cat start_broadcast.sh

#!/bin/sh

BITRATE=320
NAME="NAS $1"
STATION_DIR="/nas/Stations/lists/"
PASSWORD="secret"

AUDIO_OPT='do_audio=1:plugin=i_audiofile{album='$STATION_DIR$1':shuffle=1}'
VIDEO_OPT="do_video=0"
METADATA_OPT="metadata_mode=input"
ENC_OPT='audio_encoder=b_lame{server=nas_ip:mount=/'$1':password='$PASSWORD':name='$NAME':cbr_bitrate='$BITRATE'}'

gmerlin-record -aud $AUDIO_OPT -vid $VIDEO_OPT -m $METADATA_OPT -enc "$ENC_OPT" -r 2>> /dev/null >> /dev/null &
echo $! > $1.pid

If you call the script with start_broadcast.sh foo, it will load the album /nas/Stations/lists/foo and send the stream to the icecast server, which will make it available under nas.ip:8000/foo. In addidion, the PID of the process will be written to ./foo.pid so it can be stopped later.

The foo broadcast can be stopped with stop_broadcast.sh foo, where the
script looks like:

#!/bin/sh
kill -9 `cat $1.pid`
rm -f $1.pid

Icecast configuration
No critical options had to be changed in the icecast configuration, except queue-size, which was doubled to 1048576 because it's better for 320 kbps streams.

Icecast stats in awk friendly format
For the on-demand meachism described below, we also need to get the
running channels and connected clients from the server ideally in an awk friendly
format. This is done by getting the server statistics in xml format and process it
with xsltproc, a small commandline tool which comes with libxml2:

$cat get_stats.sh

#!/bin/sh
wget --user=admin --password=secret -O - http://127.0.0.1:8000/admin/stats.xml 2> /dev/null | \
xsltproc stats.xsl - | cut -b 2-

If you have two channels foo (1 listener) and bar (2 listeners) it will output


foo 1
bar 2

The transformation file stats.xsl looks like:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="text"/>
<xsl:template match="/">
 <xsl:for-each select="icestats/source">
    <xsl:value-of select="@mount"/>
    <xsl:text> </xsl:text>
    <xsl:value-of select="listeners"/>
    <xsl:text>
</xsl:text>
 </xsl:for-each>
</xsl:template>
</xsl:stylesheet>

On demand mechanism
Now since we have commands for starting, stopping and querying channels, we can start a channel when the first listener connects and stop it after the last listener disconnected. Since icecast doesn't support on demand streaming, we must trick it into doing so. The idea is to put a second http server in front of the icecast server, which handles the connection requests, starts the channel (if necessary) and then does a http redirect to the real icecast url. The icecast server runs on port 8000, the redirection server (to which the listeners connect) runs on port 8001. The redirection server can be built simply within shell scripts using the netcat (traditional) utility. The server script is simple:

$cat server.sh
#!/bin/sh

cd /nas/mmedia/Stations

while true; do
nc.traditional -l -p 8001 -c ./handle.sh
done

Whenever a TCP connection on port 8001 arrives, the following handler script is executed:

$cat handle.sh
#!/bin/bash

# Read request, path and protocol
read REQ URLPATH PROTO
# Read header variables
while true; do
read VAR VAL
if test "x$VAL" = "x"; then
break
fi
done

# Reject anything but GET requests
if test "x$REQ" != "xGET"; then
echo -e "HTTP/1.1 400 Bad Request\r\n\r\n"
exit
fi

# Remove leading "/"
FILE=`echo $URLPATH | cut -b 2-`

# Close unused streams
./clean.sh $FILE

# Check if we are broadcasting already
RESULT=`./query_station.sh $FILE`
if test "x$RESULT" = "x"; then
./start_broadcast.sh $FILE 2>> /dev/null &
sleep 1
fi

# Send redirection header
URL="http://nas_ip:8000/$FILE"
echo -e "HTTP/1.1 307 Temporary Redirect\r\nLocation: $URL\r\n\r\n"

Here we use 2 additional scripts. clean.sh stops all streams with zero listeners except the one, which was given as commandline argument.

#!/bin/sh
./get_stats.sh | awk -v NAME=$1 '($1 != NAME) && ($2 == 0) { system("./stop_broadcast.sh " $1) }'

query_station.sh lists just the number of listeners of the given station:

#!/bin/sh
./get_stats.sh | awk -v NAME=$1 '$1 == NAME { print $2 }'

Energy saving mode
When we just use the radio, the NAS must be switched on manually. The PCs do that automatically with wake-on-lan. The NAS detects, when it is no longer needed and switches off automatically then. This is done by querying the TCP connections to IP addresses other than localhost. If we don't have any external connections for more than 30 minutes, we switch off. The following script can be interesting for many other applications as well. Simply start it during booting:

#!/bin/sh
# Switch off after this time
THRESHOLD=1800
# Delay between 2 checks
DELAY=60

DATE_START=`date +%s`

while :
do
  CONNECTIONS=`netstat -tn | grep tcp | grep -v " 127\." | wc -l`
  DATE_NOW=`date +%s`

  if test "x$CONNECTIONS" = "x0"; then
    DATE_DIFF=`echo "$DATE_NOW - $DATE_START - $THRESHOLD" | bc`
    if test $DATE_DIFF -gt "0"; then
      poweroff
      exit
    fi
  else
    DATE_START=$DATE_NOW
  fi
  sleep $DELAY
done

Mission accomplished.

New prereleases

2010-12-09T01:31:00.003+01:00

Lots of bugs have been fixed after the last prereleases, so here are
new ones:

http://gmerlin.sourceforge.net/gmerlin-dependencies-20101209.tar.bz2
http://gmerlin.sourceforge.net/gmerlin-all-in-one-20101209.tar.bz2

The good news is, that no new features were added so the code can stabilize better.

Please test this and report any problems.

The final gmerlin release is expected by the end of the year

gmerlin prereleases

2010-09-18T17:09:00.000+02:00

Some time has gone since the last prereleases, and a lot of bugs have been
fixed since then. So here is another round:

http://gmerlin.sourceforge.net/gmerlin-dependencies-20100918.tar.bz2
http://gmerlin.sourceforge.net/gmerlin-all-in-one-20100918.tar.bz2

Gmerlin configuration improvements

2010-08-07T02:10:00.006+02:00

Up to now, gmerlins configuration philosophy was simple: Export all user settable parameters as possible to the frontends, no matter now important they are. There are 2 reasons for that:

As a developer, I don't like to decide which configuration options are important.
As a user I (personally) want to have full control over all program-and plugin settings. Nothing annoys me more in other applications than features, which could easily achieved by the backend, but they are not supported in the frontend.

The downside of this approach is simple: For an average user, the gmerlin applications are way too complicated. And of course, for me it's also annoying to tweak that many parameters all the time. Now, since I reach a one-zero version, It's time to look at such usability issues.

A little look behind the GUI
A configuration dialog can contain of multiple nested sections. If you have more than one section, you see a tree structure on the right, which lets you select the section. A section contains all the configuration widgets you can see at the same time. Therefore the code must always distinguish if an action is for a section or for the whole dialog.

Factory defaults
Most configuration sections now have a button Restore factory defaults. It does, what the name suggests. You can use this if you think you messed something up.

Presets
Some configuration sections support presets. You can save all parameters into a file and load them again after. In some situations, presets are per section. In this case you see the preset menu below the parameter widgets. If the presets are global for the whole dialog window, you see the menu below the tree view. The next image shows a single-section dialog with the preset menu next to the restore button.

The next image shows a dialog with multiple sections. The preset menu is for the whole dialog, the restore button is for the section only.

The presets are designed such, that multiple applications can share them. E.g. an encoding setup configured in the transcoder can be reused in the recorder etc. Presets are available for:

All plugins (always global for the whole plugin)
Whole encoding setups
Filter chains

There is no reason, not to support presets for other configurations as well. Suggestions are welcome.

Gmerlin prereleases

2010-08-03T20:43:00.002+02:00

gmerlin prereleases can be downloaded here:

http://gmerlin.sourceforge.net/gmerlin-dependencies-20100803.tar.bz2

http://gmerlin.sourceforge.net/gmerlin-all-in-one-20100803.tar.bz2

Highlights of this development iteration:

Pass-through of compressed streams in the transcoder
Native matroska demuxer with sample accurate seeking in webm files
VP8 decoding via ffmpeg
Lots of work on precise timing and sample accurate seeking
Presets in the GUI configuration (will blog later about that)
The gmerlin package reaches version one-zero

Please test this as much as possible and report any problems.

Getting serious with sample accuracy

2010-08-01T14:30:00.008+02:00

gmerlin-avdecoder has a sample accurate seek API for some time now. What was missing was a test to prove that seeking happens really with sample accuracy.

Test tool
The strictest test if a decoder library can seek with sample accuracy is to seek to a position, decode a video frame or a bunch of audio samples. Compare these with the frame/samples you get if you decode the file from the beginning. Of course, the timestamps must also be identical. A tool, which does this, is in tests/seektest.c. I noticed, that video streams easily pass this test, usually even if no sample accurate access was requested. That's probably because I thought, that video streams are more difficult. So I put more brainload into them. Therefore I'll concentrate on audio streams in this post.

Audio codec details
When seeking in video streams, you have keyframes, which tell you where decoding of a stream can be resumed after a seek. It's sometimes difficult to implement this, but at least you always know what to do.

The naive approach for audio streams is to assume, that all blocks (e.g. 1152 samples for mp3) can be decoded independently. Unfortunately, reality is a bit more cruel:

Bit reservoir
This is a mechanism, which allows to make pseudo VBR in a CBR stream. If a frame can be encoded with fewer bits than allocated for the frame, it can leave the remaining bits to a subsequent (probably more complex) frame. The downside of this trick is, that after a seek, the next frame might need bits from previous frames to be fully decoded.

Oberlapping transform
Most audio compression techniques work in the frequency-domain, so between the audio signal and the compression stage, there is some kind of fft-like transform.

Now, for reasons beyond this post, overlapping transforms are used by some codecs. This means, that for decoding the first samples of a compressed block, you need the last samples of the previous block. The image below shows one channel of an AAC stream for the case that the overlapping was ignored when seeking. You see that the beginning of the frame is not reconstructed properly, because the previous frame is missing.

Both the bit reservoir and the overlapping can be boiled down to a single number, which tells how many sample before the actual seek point the decoder must restart decoding. This number is set by the codec during initialization, and it's used when we seek with sample accuracy.

Mysterious liba52 behavior
Even if sample accuracy was achieved, the AC3 streams (which are on DVDs or in AVCHD files) don't achieve bit exactness. The image below shows, that there is no time shift between the signals (which means that gmerlin-avdecoder seeks correctly), but the values are not exactly the same.

First I blamed the AC3 dynamic range control for this behavior. Dynamic range compressors always have some kind memory across several frames. But even after disabling DRC, the difference was still there. I would really be curious if that's a principal property of AC3 being non-deterministic or if it's a liba52 bug.

Conclusions
The table below lists all audio codecs, which were taken into consideration. They represent a huge percentage of all files found in the wild. The next important codecs are the uncompressed ones, but these are always sample accurate.

Compression	Library	Overlap	Bit reservoir	Bit exact
MPEG-1, layer II	libmad	-	-	+
MPEG-1, layer III	libmad	+	+	+
AAC	faad2	+	? (assumed -)	+
AC3	liba52	+	? (assumed -)	- (see image above)
Vorbis	libvorbis	+	-	+

Obtaining the information summarized here was a very painful process with web researches and experiments. The documentation of the decoder libraries regarding sample accurate and bit exact seeking is extremely sparse if not non-existing.

Processing compressed streams with gmerlin

2010-05-01T02:19:00.004+02:00

As I already mentioned, a main goal of this development cycle is to read compressed streams on the input side and write compressed streams on the encoding side. It's a bit of work, but it's definitely worth it because it offers enormous possibilities:

Lossless transmultiplexing from one container to another
Adding/removing streams of a file without recompressing the other streams.
Lossless concatenation of compressed files
Changing metadata of files (i.e. mp3/vorbis tagging)
Quicktime has some codecs, which correspond to image formats (png, jpeg, tiff, tga). Supporting compressed frames can convert single images to quicktime movies and back
In some cases broken files can be fixed as well

General approach
To limit the possibilities of creating broken files, we are a bit strict about the
codecs we support for compressed I/O. This means, that with the new feature you cannot automatically transfer all compressed streams. For compressed I/O the following conditions
must be met:

The precise codec must be known to gavl. While for decoding it never matters if we have MPEG-1 or MPEG-2 video (libmpeg2 decodes both), for compressed I/O it must be known.
For some codecs, we need other parameters like the bitrate or if the stream contains B-frames or field pictures.
Each audio packet must consist of an independently decompressable frame and we must know, how many uncompressed samples are contained.
For each video packet, we must know the pts, how long the frame will be displayed and if it's a keyframe.

Compression support in gavl
For transferring compressed packets, we need 2 data structures:

An info structure, which describes the compression format (i.e. the codec). The actual codec is an enum (similar to ffmpegs CodecID), but other parameters can be required as well (see above).
A structure for a data packet.

Both of these are in gavl in a new header file gavl/compression.h. Gavl itself never messes around with the contents of compressed packets, if just provides some housekeeping functions for packets and compression definitions. The definitions were moved here, because it's the only common dependency of gmerlin and gmerlin-avdecoder and I didn't want to define that twice.

gmerlin-avdecoder
There are 2 new functions for getting the compression format of A/V streams:

int bgav_get_audio_compression_info(bgav_t * bgav, int stream,
                               gavl_compression_info_t * info)

int bgav_get_video_compression_info(bgav_t * bgav, int stream,
                               gavl_compression_info_t * info)

They can be called after the track was selected with bgav_select_track(). If the demuxer doesn't meet the above goals for a stream it's tried with a parser. If there is no parser for this stream, compressed output fails and the functions return 0.

If you decided to read compressed packets from a stream, pass BGAV_STREAM_READRAW to bgav_set_audio_stream() or bgav_set_video_stream(). Then you can read compressed packets with:

int bgav_read_audio_packet(bgav_t * bgav, int stream, gavl_packet_t * p);

int bgav_read_video_packet(bgav_t * bgav, int stream, gavl_packet_t * p);

There is a small commandline tool bgavdemux, which writes the compressed packets to raw files, but only if the compression supports a raw format. This is e.g. not the case for vorbis or theora.

libgmerlin
In the gmerlin library, the new feature shows up mainly in the plugin API. The input plugin (bg_input_plugin_t) got 4 new functions, which have the identical meaning as their counterparts in gmerlin-avdecoder:

int (*get_audio_compression_info)(void * priv, int stream,
                              gavl_compression_info_t * info);

int (*get_video_compression_info)(void * priv, int stream,
                              gavl_compression_info_t * info);

int (*read_audio_packet)(void * priv, int stream, gavl_packet_t * p);

int (*read_video_packet)(void * priv, int stream, gavl_packet_t * p);

On the encoding side, there are 6 new functions, which are used for querying if compressed writing is possible, adding compressed A/V tracks and writing compressed A/V packets:

int (*writes_compressed_audio)(void * priv,
                           const gavl_audio_format_t * format,
                           const gavl_compression_info_t * info);

int (*writes_compressed_video)(void * priv,
                           const gavl_video_format_t * format,
                           const gavl_compression_info_t * info);

int (*add_audio_stream_compressed)(void * priv, const char * language,
                               const gavl_audio_format_t * format,
                               const gavl_compression_info_t * info);

int (*add_video_stream_compressed)(void * priv,
                               const gavl_video_format_t * format,
                               const gavl_compression_info_t * info);

int (*write_audio_packet)(void * data, gavl_packet_t * packet, int stream);

int (*write_video_packet)(void * data, gavl_packet_t * packet, int stream);

gmerlin-transcoder
In the gmerlin transcoder you have a configuration for each A/V stream:

The options for the stream can be "transcode", "copy (if possible)" or "forget". Copying of a stream is possible if the following conditions are met:

The source can deliver compressed packets
The encoder can write compressed packets of that format
No subtitles are blended onto video images

All filters are however completely ignored. You can configure any filters you want, but when you choose to copy the stream, none of them will be applied.

If a stream cannot be copied, it will be transcoded.

libquicktime
Another major project was support in libquicktime. It's a bit nasty because libquicktime codecs do tasks, which should actually be done by the (de)multiplexer. In practice this means that compressed streams have to be enabled for each codec and container separately. The public API is in compression.h. It was modeled after the functions in libgmerlin, but the definition of the compression (lqt_compression_info_t) is slightly different because inside libquicktime we can't use gavl.

I made a small tool lqtremux. It can either be called with a single file as an argument, in which case all A/V streams are exported to separate quicktime files. If you pass more than one file on the commandline, the last file is considered the output file and all tracks of all other files are multiplexed into the output file. Note that lqtremux is a pretty dumb application, which was written mainly as a demonstration and testbed for the new functionality. In particular you cannot copy some tracks while transcoding others. For more sophisticated tasks use gmerlin-transcoder or write your own tool.

Status and TODO
Most major codecs and containers work, although not all of them are heavily tested. Therefore I cannot guarantee, that files written that way will be compatible with all other decoders. Future work will be testing, fixing and supporting more codecs in more containers. Of course any help (like bugreports or compatibility testing on windows or OSX) is highly appreciated.

With this feature my A/V pipelines are ready for a 1.x version now.

libtheora vs. libschrödinger vs. x264

2010-03-20T15:59:00.009+01:00

Doing a comparison of lossy codecs was always on my TODO list. That's mostly because of the codec-related noise I read (or skip) on mailing lists and propaganda for the royality free codecs with semi-technical arguments but without actual numbers. A while ago I made some quick and dirty PSNR tests, where x264 turned out to be the clear winner. But recently we saw new releases of libtheora and libschrödinger with improved encoding quality, so maybe now is the time to investigate things a bit deeper.

Specification vs implementation
With non-trivial compression techniques (and all techniques I tried are non-trivial) you must make a difference between a specification and an implementation. The specification defines how the compressed bitstream looks like, and suggests how the data can be compressed. I.e. it specifies if motion vectors can be stored with subpixel precision or if B-frames are possible. The implementation does the actual work of compressing the data. It has a large degree of freedom e.g. it lets you choose between several motion estimation methods or techniques for quantization or rate control. If you fully read and understood all specifications, you can make a rough estimation, which specification allows more powerful compression. But if you want numbers, you can only compare implementations.

This implies, that statements like "Dirac is better than H.264" (or vice versa) are inherently idiotic.

Candidates

libschroedinger-1.0.9
libtheora-1.1.0
x264 git version from 2010-03-19

Rules of the game
If compression algorithms are completely different, it's not easy to find comparable codec parameters. Some codecs are very good for VBR encoding but suck when forced to CBR. Some codecs are optimized for low-bitrate, others are work better at higher bitrates. Therefore I decided for very simple test rules:

All codec settings are set to their defaults found in the sourcecode of the libraries. This leaves the decision of good parameters to the developers of the libraries. I upgraded the codec parameters in libquicktime and the gmerlin encoding plugins for the newest library versions.
The only parameter, which is changed, corresponds to the global quality of the compression (all libraries have such a parameter). Multiple files are encoded with different quality settings.
From the encoded files the average bitrate is calculated and the quality (PSNR and MSSIM) is plotted as a function of the average bitrate.

Footage
Some lossless sequences in y4m format can be downloaded from the xiph site. I wanted a file, which has fast global motion as well as slower changing parts. Also the uncompressed size shouldn't be too large to keep the transcoding- and analysis time at a reasonable level. Therefore I decided to use the foreman. Of course for a better estimation you would need much more and longer sequences. Feel free to repeat the experiment with other files and tell about the results.

Analysis tool
I wrote a small tool gmerlin_vanalyze, which is called with the original and encoded files as only arguments. It will then output something like:

0 33385 47.137417 0.992292
1 17713 45.936294 0.989990
2 17693 45.998659 0.990233
3 17361 46.008802 0.990297
4 19253 46.144632 0.990582
5 19005 46.179699 0.990648

....

295 24454 45.174282 0.993100
296 23841 44.653152 0.992318
297 20966 43.848303 0.991013
298 13941 41.996157 0.987494
299 11682 41.852630 0.987321
# Average values
# birate      PSNR      SSIM
# 4941941.26  46.075177 0.991434

Each line consists of:

Frame number
Compressed size of this frame in bytes
Luminance PSNR in dB of this frame
Mean SSIM of this frame

The summary at the end consists of the total video bitrate (bits/second) as well as PSNR and SSIM averaged over all frames.

You can get this tool if you upgrade gavl, gmerlin and gmerlin-avdecoder from CVS. It makes use of a brand new feature (extracting compressed frames), which is needed for the video bitrate calculation (i.e. without the overhead from the container).

Results
See below for the PSNR and MSSIM results for the 3 libraries.

Conclusion
The quality vs. bitrate differences are surprisingly small. While x264 still wins, the royality free versions lag behind by just 2-3 dB of PSNR. Also surprising is that libtheora and libschroedinger are so close together given the fact, that Dirac has e.g. B-frames, while theora has just I- and P-frames. Depending on your point of view, this is good news for libtheora or bad news for libschroedinger

Another question is of course, if this comparison is completely fair. A further project could now be to take the codecs and tweak single parameters to check, how the quality can be improved. Also one might add other criteria like encoding-/decoding speed as well. Making tests with different types of footage would also give more insight.

To summarize that, I don't state that these numbers are the final wisdom. But at least they are numbers, and neither propaganda nor marketing.

AVCHD timecode update

2010-02-06T14:54:00.003+01:00

After this post I got some infos from a cat, which helped me to understand the AVCHD metadata format much better. This post summarizes, what I currently know.

AVCHD metadata are stored in an SEI message of type 5 (user data unregistered). These messages start with a GUID (indicating the type of the data). The rest is not specified in the H.264 spec. For AVCHD metadata the data structure is as follows:

1. The 16 byte GUID, which consists of the bytes

0x17 0xee 0x8c 0x60 0xf8 0x4d 0x11 0xd9 0x8c 0xd6 0x08 0x00 0x20 0x0c 0x9a 0x66

2. 4 bytes

0x4d 0x44 0x50 0x4d

which are "MDPM" in ASCII.

3. One byte, which specifies the number of tags to follow

4. Each tag begins with one byte specifying the tag type followed by 4 bytes of data.

The date and time are stored in tags 0x18 and 0x19.

Tag 0x18 starts with an unknown byte. I saw values between 0x02 and 0xff in various files. It seems however that it has a constant value for all frames in a file. The 3 remaining bytes are the year and the month in BCD coding (0x20 0x09 0x08 means August 2009).

The 4 bytes in tag 0x19 are the day, hour, minute and second (also BCD coded).

There are more informations stored in this SEI message, check here for a list.

If you want to make further research on this, you can download gmerlin-avdecoder from CVS, open the file lib/parse_h264.c and uncomment the following line (at the very beginning):

// #define DUMP_AVCHD_SEI

Then you can use bgavdump on your files. It will decode the first 10 frames from the file. If you want to decode e.g. 100 frames, use

bgavdump -nf 100 your_file.mts

Video quality characterization techniques

2010-01-27T23:24:00.014+01:00

When developing video processing algorithms and tuning them for quality, one needs proper measurement facilities, otherwise one will end up doing voodoo. This post introduces two prominent methods for calculating the differences of two images (the "original" and the "reproduced" one) and get a value, which allows to estimate, how well the images coincide.

PSNR
The most prominent method is the PSNR (peaked signal-to-noise ratio). It is based on the idea, that the reproduced image consists of the original plus a "noise signal". The noise level can be characterized by the signal-to-noise ratio and is usually given in dB. Values below 0 dB mean that the noise power is larger than the signal. For identical images (zero noise), the PSNR is infinite.

Advantage is, that it's a well established method and the calculation is extremely simple (see here for the formula). Disadvantage is, that it is a purely mathematical calculation of the noise power, while the human psychovisual system is completely ignored.

Thus, one can easily to make 2 images, which have different types of compression artifacts (i.e. from different codecs) and have the similar PSNR compared to the original. But one looks much better than the other. Therefore, current opinion among specialists is, that PSNR can be used or optimizing one codec, while it fails for comparing different codecs. Unfortunately, many codec comparisons in the internet still use PSNR.

SSIM
SSIM (structural similarity) was fist suggested by Zhou Wang et.al. in the paper "Image Quality Assessment: From Error Visibility to Structural Similarity" (IEEE Transactions on image processing, Vol. 13, No. 4, April 2004, PDF).

The paper is very well written and I recommend anyone, who is interested, to read it. In short: The structural similarity is composed of 3 values:

Luminance comparison
Contrast comparison
Structure comparison

All these components are normalized such that they are 1.0 for identical images. The SSIM index is the product of the 3 components (optionally raised by an exponent). Obviously the product will be normalized as well.

A difference to PSNR is, that the SSIM index for a pixel is calculated by taking the surrounding pixels into account. It calculates some characteristic numbers known from statistics: The mean value, the standard deviation and correlation coefficient.

One problem with the SSIM is, that the algorithm has some free parameters, which are slightly different in each implementation. Therefore you should be careful when comparing your results with numbers coming from a different routine. I took the parameters from the original paper, i.e. K1 = 0.01, K2 = 0.03 and an 11x11 Gaussian window with a standard deviation of 1.5 pixels.

Implementations
Both methods are available in gavl (SSIM only in CVS for now), but their APIs are slightly different. To calculate the PSNR, use:

void gavl_video_frame_psnr(double * psnr,
                         const gavl_video_frame_t * src1,
                         const gavl_video_frame_t * src2,
                         const gavl_video_format_t * format);

The src1, src2 and format arguments are obvious. The result (already in dB) is returned in psnr for each component. The order is RGB(A), Y'CbCr(A) or Gray(A) depending on the pixelformat. PSNR can be calculated for all pixelformats, but usually one will use a Y'CbCr format and take only the value for the Y' component. In all my tests the PSNR values for chrominance were much higher, so the luminance PSNR is the most pessimistic (i.e. most honest) value.

For SSIM you can use:

int gavl_video_frame_ssim(const gavl_video_frame_t * src1,
                        const gavl_video_frame_t * src2,
                        gavl_video_frame_t * dst,
                        const gavl_video_format_t * format);

The arguments src1, src2 and format are the same as for PSNR. The pixelformat however must be GAVL_GRAY_FLOAT, implying that only the luminance is taken into account. This decision was made after the experiences with PSNR. The SSIM indices for each pixel is then returned in dst, which must be created with the same format. The MSSIM (mean SSIM) for the whole image can then be obtained by averaging the SSIM values over all pixels. The function returns 1 if the SSIM could be calculated or 0 if the pixelformat was not GAVL_GRAY_FLOAT or the image is smaller than the 11x11 window.

It never matters which image is passed in src1 and which in src2 because both algorithms are symmetric.

Example
Below you see 11 Lena images compressed with libjpeg at quality levels from 0 to 100 along with their PSNR, SSIM and file size:

Quality: 0, PSNR: 23.54 dB, SSIM: 0.6464, Size: 2819 bytes

Quality: 10, PSNR: 29.84 dB, SSIM: 0.8473, Size: 4305 bytes

Quality: 20, PSNR: 32.74 dB, SSIM: 0.9084, Size: 5890 bytes

Quality: 30, PSNR: 34.38 dB, SSIM: 0.9331, Size: 7376 bytes

Quality: 40, PSNR: 35.44 dB, SSIM: 0.9460, Size: 8590 bytes

Quality: 50, PSNR: 36.31 dB, SSIM: 0.9549, Size: 9777 bytes

Quality: 60, PSNR: 37.16 dB, SSIM: 0.9612, Size: 11101 bytes

Quality: 70, PSNR: 38.34 dB, SSIM: 0.9688, Size: 13034 bytes

Quality: 80, PSNR: 40.00 dB, SSIM: 0.9768, Size: 16410 bytes

Quality: 90, PSNR: 43.06 dB, SSIM: 0.9863, Size: 24308 bytes

Quality: 100, PSNR: 58.44 dB, SSIM: 0.9993, Size: 94169 bytes

With these numbers I made a plot, which shows the PSNR and SSIM as a function of the JPEG quality:
The JPEGs have the most visible differences for qualities between 0 and 40. In this range the SSIM curve has the largest gradient. Above 40 (where the visual quality doesn't change much), the SSIM becomes more or less linear and reaches almost 1 for the best quality.

The PSNR curve is a bit misleading. It has the steepest gradient for the highest quality. This is understandable because PSNR would become infinite for the lossless (perfect quality) case. It has however not much to do with the subjective impression. PSNR however is better for fine-tuning codecs at very high quality levels. That's because the PSNR values will change more strongly, while SSIM will always be almost one.

Now I have the proper tools to make a comparison of different video codecs.

Disabling the X11 screensaver from a client

2010-01-24T02:53:00.004+01:00

One of the most annoying experiences when watching videos with friends on cold winter evenings is, when the screensaver starts. Media players therefore need a way to switch that off by one or several means.

The bad news is, that there is no official method for such a trivial task, which works on all installations. In addition, there is the energy saving mode, which has nothing to do with the screensaver, and must thus be disabled separately.

The Xlib method
You get the screensaver status with XGetScreenSaver(), disable it with XSetScreenSaver() and restore it after video playback. Advantage is, that this method is core X11. Disadvatage is, that it never works.

Old gnome method
Older gnome versions had a way to ping the screensaver by executing the command:

gnome-screensaver-command --poke > /dev/null 2> /dev/null

Actually pinging the screensaver (which resets the idle timer) is a better method, because it restores the screensaver even if the player got killed (or crashed). The bad news is, that starting with some never gnome version (don't know exactly which), this stopped working. To make things worse, the command is still available and even gives zero return code, it's just a noop.

KDE
I never owned a Linux installation with KDE. But with a little help from a friends, I found a method. My implementation however is so ugly, that I won't show it here :)

The holy grail: Fake key events
After the old gnome variant stopped working for me, I finally found the XTest extension. It was developed to test XServers. I abuse it to send fake key events, which are handled indentically to real keystrokes. They will reset the idle counters of all screensaver variants, and will also disable the energy saving mode.

Also it's a ping approach (with the advantage described above). But it works with an X11 protocol request instead of forking a subprocess, so the overhead will be much smaller. The documentation for the XTest extension is from 1992, so I expect it to be present on all installations, which are sufficiently new for video playback.

Here is how I implemented it:

1. Include <X11/extensions/XTest.h>, link with -lXtst.

2. Test for presence of the XTest extension with XTestQueryExtension()

3. Get the keycode of the left shift key with XKeysymToKeycode()

4. Each 40 seconds, I press the key with

XTestFakeKeyEvent(dpy, keycode, True, CurrentTime);

5. One video frame later, I release the key with

XTestFakeKeyEvent(dpy, keycode, False, CurrentTime);

The one frame delay was done to make sure, that the press and release events will arrive with different timestamps. I don't want to know, what happens if press- and release-events for a key have identical timestamps.

This method will hopefully work forever, no matter what crazy ideas the desktop developers get in the future. Also, this is one more reason not to use any GUI toolkit for video playback. If you use Xlib and it's extensions, you have full access to all available features of X. When using a toolkit, you have just the features the toolkit developers think you deserve.

Gmerlin release on the horizon

2010-01-19T00:54:00.002+01:00

New gmerlin prereleases are here:

http://gmerlin.sourceforge.net/gmerlin-dependencies-20100119.tar.bz2

http://gmerlin.sourceforge.net/gmerlin-all-in-one-20100119.tar.bz2

Changes since the last public release are:

Great player simplification. Most changes are internal, but the user should notice much faster seeking. Also the GUI player now updates the video window while the seek-slider is moved.
Simplification of the plugin configuration: In many places, where we had an extra dialog for configuring plugins, it was merged with the rest of the configuration.
A new recorder application which records audio (with OSS, Pulseaudio, Alsa, Jack and ESound) and video (with V4L1, V4L2 or the new X11 grabber). Output can be written into files or broadcasted.
A new encoding frontend (used by the recorder and transcoder), which allows more consistant and unified configuration of encoding setups.
The video window of the GUI player has now a gmerlin icon and is grouped together with the other windows.
Like always: Tons of fixes, optimizations and smaller cleanups in all packages. Got another 18% speedup when building AVCHD indexes for example.

This will be the last gmerlin release of the 0.3.X series. There are just 2 major features left, which keep me from releasing 1.0.0:

Support for compressed A/V frames in the architecture. This should allow lossless transmultiplexing with the transcoder.
Support for configuration presets: This will make using gmerlin applications much easier. The presets can be shared among application, i.e. once you found a good encoding preset, you can use it both in the transcoder and the recorder.

Please test it as much as you can and sent problem reports to the gmerlin-general list.

Flash-free live web video solution

2009-12-14T23:43:00.007+01:00

Some time ago, we all knew that video on the web was equivalent to the proprietary Flash technology. Also I used to say, that there might be political or psychological reasons for using Ogg/Theora, but never technical ones.

Well, the conditions have changed recently so it's time to bring an update on this.

HTML 5 video tag
The HTML 5 draft supports a <video> tag for embedding a video into a webpage as a native html element (e.g. without a plugin). Earlier versions of the draft even recommended that browsers should support Ogg/Theora as a format for the video. The Ogg/Theora recommendation was then removed and a lot of discussion was started around this. This wikipedia article summarizes the issue. Nevertheless, there are a number of browsers supporting OggTheora video out of the box, among these is Firefox-3.5.

Cortado plugin
In a different development, the Cortado java applet for theora playback was written. It is GPL licensed and you can just download it and put it into your webspace.

Now the cool thing about the <video> tag is, that browsers, which don't know about it, will display the contents between <video> and </video>, so you can include the applet code there. Researching a bit about the best way to do this, I read that the (often recommended) <applet> mechanism is not valid html. A better solution is here.

Webmasters side
Now if you have the live-stream at http://192.168.2.2:8000/stream.ogg your html page will look like:

<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
</head>
<body>
<h1>Test video stream</h1>
<video tabindex="0"
src="http://192.168.2.2:8000/stream.ogg"
controls="controls"
alt="Test video stream"
title="Test video stream"
width="320"
height="240"
autoplay="true"
loop="nolooping">
<object type="application/x-java-applet"
   width="320" height="240">
<param name="archive" value="cortado.jar">
<param name="code" value="com.fluendo.player.Cortado.class">
<param name="url" value="http://192.168.2.2:8000/stream.ogg">
<param name="autoplay" value="true">
<a href="http://192.168.2.2:8000/stream.ogg">Test video stream</a>
</object>
</video>
</body>
</html>

Note that for live-streams the "autoplay" option should be given. If not, firefox will try to load the first image (automatically anyway) to show it in the video widget. Then it will stop downloading the live-stream until you click start. Pretty obvious that this will mess up live-streaming.

Server side
For live streaming I just installed the icecast server, which came with my ubuntu. I just changed the passwords in /etc/icecast/icecast.xml, enabled the server in /etc/default/icecast2 and started it with /etc/init.d/icecast2 start.

Upstream side
There are lots of programs for streaming to icecast servers, most of them use libshout. I decided to create a new family of plugins for gmerlin: Broadcasting plugins. They have the identical API as encoder plugins (used by the transcoder or the recorder). The only difference is, that they don't produce regular files and must be realtime capable.

Using libshout is extremely simple:

Create a shout instance with shout_new()
Set parameters with the shout_set_*() functions
Call shout_open() to actually open the connection to the icecast server
Write a valid Ogg/Theora stream with shout_send(). Actually I took my already existing Ogg/Theora encoder plugin and replaced all fwrite() calls by shout_send().

See below for the libshout configuration of the gmerlin Ogg broadcaster.

Of course, some minor details were left out in my overview, read the libshout documentation for them. As upstream client, I use my new recorder application.

The result
See below a screenshot from firefox while it plays back a live stream:

Open Issues
The live-video experiment went extremely smooth. I discovered however some minor issues, which could be optimized away:

Firefox doesn't recognize live-streams (i.e. streams with infinite duration) properly. It displays a seek-slider which always sticks at the very end. Detecting a http stream as live can easily be done by checking the Content-Length field of the http response header.
The theora encoder (1.1.1 in my case) might be faster than the 1.0.0 series, but it's still way too slow. Live encoding of a 320x240 stream is possible on my machine but 640x480 isn't.
The cortado plugin has a loudspeaker icon, but no volume control (or it just doesn't work with my Java installation)

Other news

With the video switched off I can send audio streams to my wlan radio. This makes my older solution (based on ices2) obsolete.
The whole thing of course works with prerecorded files as well. In this case, you can just put the files into your webspace and your normal webserver will deliver them. No icecast needed.

Introducing Gmerlin recorder

2009-12-06T18:08:00.004+01:00

Gmerlin-recorder is a small application, which records audio and video from hardware devices. It was written as a more generic application, which should eventually replace camelot. See below for a screenshot:

As sources, we support:

Audio devices via OSS, Alsa, Esound, Pulseaudio and Jack
Webcams via V4L and V4L2
X11 grabbing

For monitoring you see the video image in the recorder window. For the audio you have a vumeter. With recorded streams you can do lots of stuff:

Filter the streams using gmerlin A/V filters
Write the streams to files using gmerlin encoder plugins
Send the encoded stream to an icecast server (will be described in detail in another blog post)
Save snapshots to images either manually or automatically

With all these features combined in the right way, you can use gmerlin-recorder for a large number of applications:

Make vlogs with your webcam and a microphone
Digitize analog music
Make a video tutorial for your application
Start your broadcasting station
Send music from your stereo to your WLAN radio
Make stop-motion movies
...

Until the recorder can fully replace camelot, we need the following features, which are not there yet:

Fullscreen video display
Forwarding of the video stream via vloopback

X11 grabbing howto

2009-11-15T14:17:00.004+01:00

One gmerlin plugin I always wanted to program was an X11 grabber. It continuously grabs either the entire root window, or a user defined rectangular area of it. It is realized as a gmerlin video recorder plugin, which means that it behaves pretty much like a webcam.

Some random notes follow.

Transparent grab window
The rectangular area is defined by a grab window, which consists only of a frame (drawn by the window manager). The window itself must be completely transparent such that mouse clicks are sent to the window below. This is realized using the X11 Shape extension:

XShapeCombineRectangles(dpy, win,
                     ShapeBounding,
                     0, 0,
                     (XRectangle *)0,
                     0,
                     ShapeSet,
                     YXBanded);

The grab window can be moved and resized to define the grabbing area. This is more convenient than entering coordinates manually. After a resize, the plugin must be reopened because the image size of a gmerlin video stream cannot change within the processing loop.

Make the window sticky and always on top
Depending on the configuration the grab window can always be on top of the other windows. There is also a sticky option making the window appear on all desktops. This is done with the collowing code, which must be called each time before the window is mapped:

if(flags & (WIN_ONTOP|WIN_STICKY))
{
Atom wm_states[2];
int num_props = 0;
Atom wm_state = XInternAtom(dpy, "_NET_WM_STATE", False);
if(flags & WIN_ONTOP)
 {
 wm_states[num_props++] =
   XInternAtom(dpy, "_NET_WM_STATE_ABOVE", False);
 }
if(flags & WIN_STICKY)
 {
 wm_states[num_props++] =
   XInternAtom(dpy, "_NET_WM_STATE_STICKY", False);
 }

XChangeProperty(dpy, win, wm_state, XA_ATOM, 32,
               PropModeReplace,
               (unsigned char *)wm_states, num_props);
}

Grabbing methods
The grabbing itself is done on the root window only. This will grab all other windows inside the grab area. The easiest method is XGetImage, but that allocates a new image with each call. malloc()/free() cycles within the processing loop should be avoided whenever possible. XGetSubImage() allows to pass an allocated image. Much better of course is XShmGetImage(). It was roughly 3 times faster than XGetSubImage() in my tests.

Coordinate correction
If parts of the grabbing rectangle are outside the root window, you'll get a BadMatch error (usually exiting the program), no matter which function you use for grabbing. You must handle this case and correct the coordinates to stay within the root window.

Mouse cursor
Grabbing works for everything displayed on the screen (including XVideo overlays) except the mouse cursor. It must be obtained and drawn "manually" onto the grabbed image. Coordinates are read with XQueryPointer(). The cursor image can be obtained if the XFixes extension is available. First we request cursor change events for the whole screen with

XFixesSelectCursorInput(dpy, root,
                    XFixesDisplayCursorNotifyMask);

If the cursor changed and is within the grabbing rectangle we get the image with

im = XFixesGetCursorImage(dpy);

The resulting cursor image is then converted to a gavl_overlay_t and blended onto the grabbed image with a gavl_overlay_blend_context_t.

Not done (yet)
Other grabbing programs deliver images only when something has changed (resulting in a variable framerate stream). This can be achieved with the XDamage extension. Since the XDamage extension is (like many other X11 extensions) poorly documented, I didn't bother to implement this yet.

One alternative is to use gmerlins decimate video filter, which compares the images in memory. The result will be the same, but CPU usage will be slightly increased.

gavl color channels

2009-10-31T16:27:00.003+01:00

When programming something completely unrelated, I stumbled across a missing feature in gavl: Extract single color channels from a video frame into a grayscale frame. The inverse operation is to insert a color channel from a grayscale frame into a video frame (overwriting the contents of that channel). This allows you to assemble an image from separate color planes.

Both were implemented with a minimalistic API (1 enum and 3 functions), which works for all 35 pixelformats. First of all we have an enum for the channel definitions:

typedef enum
{
GAVL_CCH_RED,    // Red
GAVL_CCH_GREEN,  // Green
GAVL_CCH_BLUE,   // Blue
GAVL_CCH_Y,      // Luminance (also grayscale)
GAVL_CCH_CB,     // Chrominance blue (aka U)
GAVL_CCH_CR,     // Chrominance red (aka V)
GAVL_CCH_ALPHA,  // Transparency (or, to be more precise, opacity)
} gavl_color_channel_t;

For getting the exact grayscale format for one color channel you first call:

int gavl_get_color_channel_format(const gavl_video_format_t * frame_format,
                               gavl_video_format_t * channel_format,
                               gavl_color_channel_t ch);

It returns 1 on success or 0 if the format doesn't have the requested channel. After you have the channel format, extracting and inserting is done with:

int gavl_video_frame_extract_channel(const gavl_video_format_t * format,
                                   gavl_color_channel_t ch,
                                   const gavl_video_frame_t * src,
                                   gavl_video_frame_t * dst);

int gavl_video_frame_insert_channel(const gavl_video_format_t * format,
                                 gavl_color_channel_t ch,
                                 const gavl_video_frame_t * src,
                                 gavl_video_frame_t * dst);

In the gmerlin tree, there are test programs exctractchannel and insertchannel which test the functions for all possible combinations of pixelformats and channels. They are in the gmerlin tree and not in gavl because we load and save the test images with gmerlin plugins.

Major gmerlin player upgrade

2009-10-24T01:54:00.006+02:00

While working on several optimization tasks of the player engine, I found out that the player architecture sucked. So I made a major upgrade (well, a downgrade actually since lots of code was kicked out). Let me elaborate what exactly was changed.

Below you see a block schematics of the player engine as it was before (subtitle handling is omitted for simplicity):
The audio- and video frames were read by the input thread from the file, pulled through the filter chains (see here) and pushed into the fifos.

The output threads for audio and video pulled the frames from the fifos and sent them to the soundcard or the display window.

The idea behind the separate input thread was that if CPU load is high and decoding a frame takes longer than it should, the output threads can still continue with the frames buffered in the fifos. It turned out that this was the only advantage of this approach, and it only worked if the average decoding time was still less than realtime.

The major disadvantage is, that if you have fifos with frames pushed at the input and pulled at the output, the system becomes very prone to deadlocks. If fact, the code for the fifos became bloated and messy over the time.

While programming a nice new feature (updating the video display while the seek slider is moved), the playback was messed up after seeking and I quickly blamed the fifos for this. The resulted was a big cleanup, the result is shown below:
You see, that the input thread and fifos are completely removed. Instead, the input plugin is protected by a simple mutex and the output threads do the decoding and processing themselves. The advantages are obvious:

Much less memory usage (one video frame instead of up to 8 frames in the fifo)
Deadlock conditions are much less likely (if not impossible)
Much simpler design, bugs are easier to fix

The only disadvantage is that if a file cannot be decoded in realtime, audio and video run out of sync. In the old design the input thread took care that the decoding of the streams didn't run too much out of sync. For these cases, I need to implement frame skipping. This can be done in several steps:

If the stream has B-frames, skip them
If the stream has P-frames, skip them (display only I-frames)
Display only every nth I-frame with increasing n

Frame skipping is the next major thing to do. But with the new architecture it will be much simpler to implement than with the old one.

Frame tables

2009-09-25T00:04:00.003+02:00

When thinking a bit about video editing, I thought it might be nice to have the timestamps of all video frames in advance, i.e. without decoding the whole file. With that you can e.g. seek to the Nth frame or for a given absolute time you can find the closest video frame even for variable framerate files.

The seeking functions always take a (scaled) time as argument and I saw no reason to change that.

So what I needed was a table which allows the translation of framecounts to/from PTS values even for variable framerate files. For many fileformats a similar information is already stored internally (as a file index) so the task was only to convert the info into something usable and export it through the public API.

Since this feature is very generic and might get used in both gmerlin and gmerlin-avdecoder, I decided to put the stuff into gavl.

Implicit table compression
One very smart feature in the Quicktime format is the stts atom. That's because it stores a table of frame_count/frame_duration pairs. The nice thing is, that for constant framerate files (the vast majority) the table consists of just one entry and translating framecounts to/from timestamps becomes trivial. Only for variable framerate streams, the table has more entries and the translation functions need longer.

The implementation
The frame table is called gavl_frame_table_t and is defined in gavl.h. Note: The structure is public at present but might become private before the next public release.
Here, there are also the translation functions gavl_frame_table_frame_to_time(), gavl_frame_table_time_to_frame() and gavl_frame_table_num_frames().

If you use gmerlin-avdecoder for decoding files, you can use bgav_get_frame_table() to obtain a frame table of a video stream. It can be called after the stream has been fully initialized. Naturally you will want to use sample accurate decoding mode before obtaining the frame table. If you want to reuse the frame table after the file was closed, use gavl_frame_table_save() and gavl_frame_table_load().

Future extension
An also interesting information are the timecodes. First of all, what's the difference between timecodes and timestamps in gmerlin terminology? Timestamps are used to synchronize multiple streams (audio, video, subtitles) of a file with each other for playback. They usually start at zero or another small value and play an important role e.g. after seeking in the files. Timecodes are usually given for each video frame and are used to identify scenes in a bunch of footage. They can resemble e.g. recording time/date.

Timecodes will also be supported by the frame table, but this isn't implemented yet.

Back from Malta

2009-09-15T01:14:00.003+02:00

After a number of adventure- and/or conference trips, I decided to make a more ordinary vacation. Since the goal was to extend the summer a bit, we decided to travel in the first 2 September weeks (when German weather can already start to suck) and move to the southernmost country, which can be reached without many difficulties. Easy traveling for Europeans means to stay within the EU (and Euro zone), so the destination was chosen to be Malta.

I knew a bit about Malta from TV documentaries, and while reading the Wikipedia article a bit, I became even more curious.

Here are some facts I figured out. For pictures click on the Wikipedia links, they have higher quality than the ones from my crappy mobile.

It was damn hot and humid. Not all people from the Northern countries can withstand that. I was happy to find out that after surviving 4 weeks in the monsoon season in Southern India, I'm more heat resistant than my Greek companion.
It is indeed very stress-free because everyone speaks English (unlike in other Southern European countries) and the islands are small enough to reach practically every destination by bus. Just make sure you stay in the capital Valletta, almost all bus routes start there. On the smaller island Gozo, all busses start in Victoria.
It has an extremely rich 7000 years old history. You can visit prehistoric temples (probably the oldest of their kind worldwide), churches, fortresses and remains from the Romans and Phoenicians. I already saw many churches from the inside before, I was not really a fan of Latin language and Roman history in school, and my companion didn't want to bother looking for sight seeing destinations at all. So I decided to concentrate on the prehistoric stuff.
We saw the temples of Ħaġar Qim, Mnajdra (both covered with giant protective tents now), Tarxien and Ġgantija. The latter ones were the most impressive, especially after we reached them after a long walk in the afternoon heat 20 minutes before the last admission :)
The temples of Skorba are the oldest ones, but not very spectacular and not accessible for the public.
No chance to get into the Hypogeum of Ħal-Saflieni. It was booked out 4 weeks in advance. That sucked.
If you like beaches, Malta is definitely not the #1 destination for you. There are just a few, a nice one we visited is Ramla Bay.
If you like diving at cliffs, Malta is a paradise for you.
Funny is the bit of British culture (Malta was British until 1964). People are driving on the left, you get chips with most meals, and the phone booths and mailboxes are red, like in the UK.
Some Maltese people know more German soccer teams than me.

Shortly after coming back home, German weather started to suck.

Quick & dirty fix for the latest linux NULL pointer vulnerability

2009-08-14T17:56:00.003+02:00

This one is pretty scary. It is the result of several flaws in SELinux, pulseaudio and some obscure network protocols. Proper fixing of this would require work at many places in the code.

Up to now, Ubuntu doesn't have a patched kernel. In the meantime, place the following into the modprobe configuration:

install appletalk /bin/true
install ipx /bin/true
install irda /bin/true
install x25 /bin/true
install pppox /bin/true
install bluetooth /bin/true
install sctp /bin/true
install ax25 /bin/true

Then either unload these modules by hand (if they are loaded) or reboot the machine. One some systems I had to uninstall bluetooth support, which wasn't needed anyway. Naturally these protocols will stop working, but fortunately the exploit will stop working either :)

Spambot ladies @ blogspot

2009-08-07T00:53:00.007+02:00

She left this heart warming comment on my blog. Out of curiosity, I wanted to see what Martha aka Susan blogs herself. Then I wondered, how a person, who isn't even able to blog one meaningful sentence, can be interested in my articles. But look, this lady seems to have an extremely wide range of interests (although she's not very imaginative in commenting):

She has multiple blogs under different names, and some people even talk to her as if she was a human.

Let's see how this develops in the future. Jack Crossfire, the original author of cinelerra and quicktime4linux (from which we forked libquicktime), also has something to say on this matter.

AVCHD timecodes revealed

2009-08-01T02:22:00.006+02:00

When playing with my new Canon HF200 camera, I got curious where the recording time (and date) is hidden in the AVCHD format.

The first idea was the SEI pic timing message of the H.264 stream. I already parse it for getting information whether pictures are frame- or field coded. So I extended my code to parse the timecode in HH:MM:SS:FF format, only to find out, that this info isn't present at all in my files :(

Googling for more informations about that, I found that nobody knows how to get the recording time and even professional programs fail to display it. But some very few programs do, so we know that it must be coded in the transport stream itself (and not in the other files written by the camera).

Finally I found this perl script, which extracts the date and time from canon mts files. It's a pretty simple implementation: It scans the multiplexed transport stream for a particular bit-pattern and then extracts the data. The script works for Canon-files but fails e.g. for Panasonic files.

Then I found where exactly the information is located: A H.264 stream has SEI (supplemental enhancement information) messages, which can contain additional (e.g. timing) information. For each SEI message the parser can obtain the message type (an integer) and the size of the message in bytes. AVCHD files have SEI messages of type 5, which means "user data unregistered" (== proprietary extension). The H.264 standard says, that these messages start with a 16 byte GUID followed by the payload.

Now take a look at the hexdump of such an SEI message:

17 ee 8c 60 f8 4d 11 d9 8c d6 08 00 20 0c 9a 66 ...`.M...... ..f
4d 44 50 4d 09 18 02 20 09 08 19 01 01 25 45 70 MDPM... .....%Ep
c7 f2 ff ff 71 ff ff ff ff 7f 00 00 65 84 e0 10 ....q.......e...
11 30 02 e1 07 ff ff ff ee 19 19 02 00 ef 01 c0 .0..............
00 00                                           ..

From this I found the following structure:

The GUID is the first 16 bytes. It's always the same for the info we want, but I found other SEI messages of type 5 with different GUIDs in AVCHD files.
4 characters "MDPM". They occur in all files I looked at.
An unknown byte (0x09, other vendors have other values)
The byte 0x18 (probably indicating that year and month follow)
An unknown byte (0x02, other vendors have other values)
The year and month in 3 BCD coded bytes: 0x20 0x09 0x08
The byte 0x19 (probably indicating that day and time follow)
An unknown byte (0x01, other vendors have other values)
The day, hour, minute and second as 4 BCD encoded bytes (0x01 0x01 0x25 0x45)

In this case, I extract the recording time "2009-08-01 01:25:45" (which is correct).

The remainder of the SEI is completely unknown, but I'm sure if someone would figure out the complete data structure (including the unknown bytes), one might be able to extract other interesting informations.

These messages are present for almost all frames, but I plan to read them only from the first frame because the following ones are redundant.

Next project will be to clean up the parsing code in gmerlin-avdecoder and make the timecode actually appear along with the first decoded frame.

GPU accelerated H.264 decoding

2009-07-20T23:45:00.003+02:00

A while ago I bought a new camera and I had to learn that my dual core machine isn't able to play footage encoded in highest quality level in realtime (second highest quality works). Fortunately, ffplay can't do that either so it's not gmerlins fault.

H.264 decoding on the graphics card can be done with vdpau or vaapi. The former is Nvidia specific and libavcodec can use it for H.264. The latter is vendor independent (it can use vdpau as backend on nvidia cards) but H.264 decoding with vaapi is not supported by ffmpeg yet.

In principle I prefer vendor independent solutions, but since I need H.264 support and ATI cards suck anyway on Linux, I tried VDPAU first.

The implementation in my libavcodec video frontend was straightforward after studying the MPlayer source. The VDPAU codecs are completely separated from the other codecs. They can simply be selected e.g. with avcodec_find_decoder_by_name("h264_vdpau"). Then, one must supply callback functions for get_buffer, release_buffer and draw_horiz_band. That's because the rendering targets are no longer frames in memory but rather handles of data-structures on the GPU. See here and here to see the details.

After the decoding, the image data is copied to memory by calling VdpVideoSurfaceGetBitsYCbCr. This brings of course a severe slowdown. A much better way would be to keep the frames in graphics memory as long as possible. But this needs to be done in a much more generic way: Images can be VDPAU or VAAPI video surfaces, OpenGL textures or whatever. Implementing generic support for video frames, which are not in regular RAM, will be another project.

Gmerlin releases

2009-07-13T23:53:00.001+02:00

Get them here.

Linux serial port driver madness

2009-07-01T18:27:00.003+02:00

Serial ports are almost extinct on desktop PCs because everyone uses USB or ethernet. But for many automation tasks, RS-232 connections are still state of the art. That's mostly because they are well standardized since the 60s and UART chips are cheap.

I wanted to write a program, which uses a serial port for controlling 2 stepper motors. I wanted to write it such that users have the least possible trouble. Especially during the startup phase, a dialog box should show all available serial ports and let the user select the right one.

And while trying to get all available serial ports, I stumbled across 3 nasty linux bugs:

Bug 1: Always 4 serial port devices
One of the intentions of udev was that only the devices, which are physically present on the system, appear as nodes in the /dev directory. A big step forward compared to the situation before. Unfortunately, the serial driver always creates /dev/ttyS[0-3]. The reason (forgot the link) is that the ports of some exotic multi I/O board aren't detected properly. So the fix was to get 1000s of users into trouble instead of just making one special module parameter for that board.

Bug 2: open() succeeds for nonexistant ports
The manual page of open(2) says, that if one tries to open a device node with no physical device is behind it, errno is set to ENODEV or ENXIO. In the case of a non existing serial port, a valid filedescriptor is returned

Bug 3: tcgetattr returns EIO
According to the glibc manual EIO is usually used for physical read/write errors. For regular files this error means, that you should backup your data because the disk is about to die. A physical read/write error can never happen on a device which is physically nonexistant.

The second best thing would be to return ENXIO (no such device or address). The best thing would be to return EBADF (bad file descriptor) because the open() call before would have returned -1 already.

At least tcgetattr() doesn't succeed like open() so it can be used for the detection routine below.

The solution
The good thing is that once there is a workaround, the problem is quickly forgotten. Here's mine (returns 1 if the port is a physically existant serial device, 0 else):

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <termios.h>
#include <unistd.h>

int is_serial_port(const char * device)
 {
 int fd, ret = 1;
 struct termios t;
 fd = open(device, O_RDWR);

 if(fd < 0)
   return 0;

 if(tcgetattr(fd, &t))
   ret = 0;
 close(fd);
 return ret;
 }

Of course this won't find ports, which are currently opened by another application.

New gmerlin prereleases

2009-06-26T00:32:00.001+02:00

Here we go:

http://gmerlin.sourceforge.net/gmerlin-all-in-one-20090626.tar.bz2
http://gmerlin.sourceforge.net/gmerlin-dependencies-20090625.tar.bz2

What to expect for the remaining century

2009-06-25T01:03:00.003+02:00

Web censorship by police agencies, terrorism of the music industry against ordinary people and other scaring stories indicate that society changes dramatically these days. These changes are in my opinion hopelessly underestimated by most. But what exactly is happening? I'll try to explain it with some examples from history.

Technological developments are triggers for social changes
It has been always like this: If a new technology appears, it offers both chances and challenges. Chances to solve some problems of mankind come always along with the challenge to use it responsibly. If you look back in history, you'll find many examples, where inventions resulted in revolutionary changes of society.

6000 BC
One of the earliest known examples for this phenomenon is the invention of irrigation around 6000 BC in Mesopotamia. It triggered the following chain reaction:

Farmland became more efficient allowing a higher population density
Higher population density results either in self-extinction, mass exodus or a better organization. The Mesopotamians were smart enough to choose the latter.
Suddenly there was some essential infrastructure (the irrigation channels), which had to be maintained. People who maintain irrigation channels cannot work on the fields at the same time, so the division of labor had to be invented
If you have division of labor, you need trade
If you have trade, you need money
If you have money, you need mathematics
Money and trade hardly work if you don't invent some alphabet

What looks like an amazingly fast transition from an archaic rural society to an urban civilization is, in my opinion, simply a result of some smart farmers no longer wanting to wait for rain.

20th century
The beginning of the 20th century was characterized by by lots of revolutionary advances especially in electronic communication (radio and TV), transportation (cars and airplanes) and manufacturing (mass production). Some of the results follow:

Ordinary people could more easily stay informed about political affairs. Without a well informed population modern democracy is impossible.
What we call a global society today, became possible through electronic communication.
Many cultural genres, e.g. music styles, are no longer local (or national) phenomena but global ones.
In years where the weather sucks (from the farmers points of view) people no longer have to be afraid of famines (today's famines always have political reasons). This feeling of safety has a large impact on human mind and society.

As you might see, the technological revolution at the beginning of the 20th century was an important trigger for the development of the so called Western Civilization we are so proud of today.

Irresponsible uses of 20th century technology were European dictatorships (which utilized the then new electronic mass media for propaganda) resulting in 2 World Wars (made possible by the newly invented cars and planes).

The biggest challenge of the 20th century was not to start a nuclear war. Before, all weapons ever developed were eventually used. The atomic bomb was never used in a war by any nation except one.

Beginning of the 21st century
The beginning of the 21st century was characterized by masses of ordinary people beginning to use the internet. Many argue, that the internet is just another means of communication like the ones we had before. This is completely wrong imo. All mass media we had before were unidirectional (few content producers serve many consumers). The internet is multidirectional (every content consumer can also be a producer). This is a completely different topology of information flow with possibilities far beyond everything we had before. Just look at some examples of what was achieved by now:

People write an encyclopedia, which is larger, more up-to-date but not more incorrect than any paper-encyclopedia you could buy before.
1000s of computer nerds spending their nights in front of their PCs wrote one of the world's best OSes.
In times of war or unrest, gagging orders no longer work
No matter what absurd theories you believe, you'll always find like-minded people.
Some internet movies become more popular than some commercial Hollywood productions.

Future developments
Some things I see coming:

Internet communities can make big achievements even though they are mostly self organized. Politicians are afraid of self organizing structures because they fear to become superfluous.
People, who work voluntarily on self-organized projects, will no longer feel the need to be ruled by politicians.
Big record companies will die out, because they terrorized their customers and ripped off musicians. Even Paul McCartney let Starbucks merchandise an album already. Also they still follow the obsolete model of few producers serving many consumers. Only small manufacturers of vinyl disks will survive (at least as long as there are DJs who know how to use them).
Traditional newspapers will become superfluous because they cost money and the 12 hour delay due to the printing time will be unacceptable for most people.
Traditional radio and TV-stations will die out because people don't want to read programming schedules anymore. They want to see and hear what they want when they want.
With no unidirectional mass media left, controlling the information flow will become impossible. As a result, Dictators and conspirators will have a hard time.

We see, that today's elites have enough reasons to try to control the internet. Actually there are 2 possible outcomes:

1. They learn that their undertaking is doomed to failure and try to arrange with the new conditions before it's too late.
2. They will continue trying and the WWW will move from today's client-server model to strongly encrypted p2p technologies like freenet.

I personally would prefer option 1, but in either case they'll lose.

Challenges
As said before, new technologies also bring new challenges:

Computer expertise becomes more and more important for ordinary people. In the last century, learning to read and write was already a good start of the career. Today, knowing to use a computer mouse is at least as important.
The fact that in the internet everyone can be content provider naturally results in a large percentage of bullshit. The preselection of information, which was done by editorial offices of the traditional mass media, must now be done by the consumer. The advantage of unfiltered information comes along with the task to decide yourself.

Even if you don't agree with my theories explained above, you know at least where the title of my blog comes from :)

If you agree with what I said, you'll like this movie.

Being ruled by morons

2009-06-19T00:29:00.005+02:00

17. June 2009: German parliament commemorates the uprising in Eastern Germany of 1953. The uprising was called the start of a fight for freedom, which was successful in the end (resulting in the German reunification in 1990). Before 1990, the 17. of June was the national holiday in Western Germany.

18. June 2009: German parliament passes a law, which allows the federal German police agency to put together a blacklist of domains, which must be blocked by ISPs. This law violates many fundamental principles of German constitution like separation of powers, federalism, freedom of information and the prohibition of censorship.

Below you'll find some preliminary success stories regarding the latest developments:

As explained before, the actual target group of this law (producers and consumers of child pornography) can be more relaxed than before, because it comes pretty handy for them. You'll see none of them demonstrating against this.
Conspiracy theorists said from the beginning, that the fight against child pornography is just a bogus argument for installing a censorship infrastructure. Fortunately, some politicians were even stupid enough to admit this in public.
Before this madness started, there was no noticeable digital civil rights movement in Germany. This changed dramatically during the last months. Now they are many and perfectly organized. On Saturday, there will be coordinated demonstrations in at least 11 German cities.
An official online petition got signed by 134014 people, an all-time record in the history of online-petitions.
The German Social Democratic Party is one big step further in making itself superfluous. I predict that some of the last bright minds will leave the party before the next parliamentary elections in September.
Rulers of China and Iran no longer have to fear bothering remarks regarding their censorship policies from German politicians. Improved bilateral relations are good for German economy, which is highly dependent on foreign sales markets and energy resources.
Almost unknown a year ago, the German Pirate Party got 0.9% in the European parliamentary elections, with constantly rising popularity.

This is the start of a fight for freedom, which will be successful in the end.

gmerlin video thumbnailer

2009-06-10T00:32:00.002+02:00

The project
The goal of this 2-day project was to write a gmerlin based video thumbnailer, which fails for fewer files than the gstreamer based thumbnailer shipped with Ubuntu. The result is a small (less than 350 lines) program, which takes a video file as argument and produces a small png preview. All required functionality was already available in gmerlin and gavl.

How to try
If you want, you can check out and install the latest gmerlin CVS. Make sure that gmerlin is compiled with libpng support. The program is called gmerlin-video-thumbnailer. To enable gmerlin thumbnails in nautilus, open gconf-editor and go to /desktop/gnome/thumbnailers. For each mimetype, you have a directory. In each directory, you have a key called command. For each mimetype you want to be thumbnailed by gmerlin, simply change the command to

/usr/local/bin/gmerlin-video-thumbnailer -s %s %i %o

Of course you must change /usr/local to the prefix where you installed gmerlin.

Bugs fixed
The thumbnailer quickly exposed some bugs in gmerlin-avdecoder and gavl. Looking at a nautilus window full of thumbnails is faster than playing 100s of movie files. 3 major bugs were fixed, one minor bug will get fixed before the next release.

References
The thumbnail standard, which seems to be used by gnome, is here. How thumbnailers are installed in gnome, is described here.

Gmerlin prereleases

2009-06-06T01:09:00.003+02:00

Check these out:

http://gmerlin.sourceforge.net/gmerlin-dependencies-20090606.tar.bz2
http://gmerlin.sourceforge.net/gmerlin-all-in-one-20090606.tar.bz2

If you want to help pushing things forward, try to install and run them and report any problems you encountered.

What evildoers do, if the web gets censored

2009-05-25T23:29:00.003+02:00

Lots of discussion happens in German cyberspace around the planned blocking of child porn. It is foreseen to redirect DNS queries for blacklisted domains to a "Stop" sign, which will tell the user, that the site (s)he wanted to visit is blocked. Contrary to earlier promises the IP addresses of the users will be made available to the law enforcement agencies for what they call real time surveillance.

The proponents (one of them confused "servers" and "surfers" in a parliament debate) try their best to make every opponent of this plan look like a pedophile.

I think it's the opposite: Child molesters and other evildoers will have a lot of fun once the blocking mechanism is operational.

Disclaimer
I neither plan nor endorse any of the activities listed below. I only describe theoretical possibilities to demonstrate, how easy it is to abuse such a nonsense.

Things to come

Pedophile newbies, who search for child porn, will find a leaked block-list from another country. The secret German list will leak as well, sooner or later. It was never as easy as today.
Less smart pedophiles will google for ways to circumvent the blockade. The first hit will be some commercial "surf anonymous" proxy provider. In the end, even more people will earn money with abused kids.
Smarter pedophiles will google better and configure another DNS server. If using another DNS server becomes illegal, they'll find a free SSL proxy. If SSL proxies become illegal, they'll use foreign VPN provider. Connections to foreign VPN providers cannot be illegal because they are used by lots of big companies.
Less smart child porn webmasters will simply use numeric IPs for linking to their sites
Smarter webmasters will regularly send a DNS query for their domain to a censored DNS server. If the returned IP number becomes incorrect, it's time to shut down the site and reopen it somewhere else under a different name. Can easily be scripted into a fully automated warning system. Also works if the block-list is really secret.
Never give your IP address to a bad guy. Today you can get hacked, if you have a poor firewall. Tomorrow the cops might pay you a visit. There is no firewall against cops wanting to visit you. That's because one can easily flood the DNS server of your German provider with forged requests for some blocked domain and your IP address as source. Should be no more than 50 lines in C, including comments.
The ultimate end of civilization is near if the first stop-sign worm appears. Not only because it'll DDOS the stop-sign servers. Most important effect will be to flood the law enforcement agencies with IPs of innocent people. Less time to concentrate on the bad guys.

Those, who know the facts and still want to install DNS blockades against child porn, are either pedophiles, criminals or conspirators who'd like a censorship mechanism for all kinds of unpleasant content. The latter might be the reason why each country maintains it's own block list.

Alternatives
Everyone except the pedophiles supports a serious fight against child abuse and child porn, not only in the internet. There are lots of organizations, who are committed to this. Interestingly, some of them are against blockades. Here are 3 German ones:

Carechild
Abuse victims against internet blockades
Self-help group of abused women

Psychedelic visualization programming howto

2009-05-10T02:42:00.003+02:00

A 3 day project was to enhance gmerlins default visualization (the one which comes with the gmerlin tarball) a bit. Here is a description, how this visual work. There are lots of other visuals out there, which are based on the same algorithm.

General alrorithm
It's actually pretty simple: Each frame is the last frame filtered a bit and with something new drawn onto it. This repeats over and over.

Drawing something
There are 2 absolutely trivial things to draw, which are directly generated by the audio signal:

Stereo waveform: Looks pretty much like a 2 beam oscilloscope
Vectorscope: One channel is the x-coordinate, the other one is the y-coordinate. The whole thing is then rotated by 45 degrees counterclockwise. It only sucks for mono signals, because it will display a pretty boring vertical line then.

The vectorscope is drawn as lines and as dots. This makes a total of 3 foreground patterns in 7 colors.

Filtering
There are 3 types of filters, which are always applied to the last frame: An image transformation, a colormatrix and a 3-tap blur filter. The blur filter is needed to wipe out artifacts when doing the image transform repeatedly on the same image.

The image transform uses the (usually bad) linear interpolation, which means that the recursive process also blurs the image. There are 8 transforms: Rotate left, rotate right, zoom in, zoom out, ripple, lens, sine and cosine.

Then there are 6 colormatrices. Each of these fades 1 or 2 of the RGB components to zero. Together with the foreground colors (which always have all 3 components nonzero) we get a large variation of the colors when they are recursively filtered.

Background colors
The image transformation sometimes makes areas in the current frame, which are outside of the last frame. These areas are filled with one of 6 background colors. Background colors are dark (all components < 0.5) to increase the contrast against the foreground.

Beat sensitivity
Then I wrote the most simple beat detection: For each audio frame, compute the sound energy (i.e. calculate the sum of the squared samples for each channel). If the energy is twice as large as the time averaged energy, we have a beat. The time averaged energy if calculated by doing a simple RC low-pass filtering of the energies for each frame:

avg_energy = avg_energy * 0.95 + new_energy * 0.05;

Variations
The following things can be changed on each beat (and a random decision with defined probability):

Foreground pattern (3)
Background color (5)
Transform (8)
Colormatrix (6)

This makes 5040 principal variations.

Result
I hacked a small tool (gmerlin_visualize), which takes an audio file and puts it into a video file together with the visualization. The result is here:

Simple music visualization from gmerlin on Vimeo.

As you can see, the video compression on vimeo isn't really suited for that kind of footage. But that's not vimeos fault. It's a general problem that video compression algorithms are optimized for natural images rather than computer generated images.

Possible improvements
The variety can easily be extended by adding new image transformations and new foreground patterns. Also, some contrast between foreground and background is sometimes a bit weak. A more careful selection of the colors and colormatrices would bring less variety but a nicer overall appearance.

Feel free to make a patch :)

Lessons from LAC

2009-04-25T19:23:00.004+02:00

I was at the Linux Audio Conference in Parma/Italy. The weather sucked most of the time, but everything else was great. These are the most important things I learned:

Jack rules
Of course I already heard about the Jack sound system before. But since I never had a linux distribution, which installs jack by default, I didn't pay much attention to it.

If you go to the LAC, you start to think that there is hardly any audio architecture besides jack. After looking a bit into Jack (and writing gmerlin plugins for Jack audio I/O) I learned, that it's actually pretty cool. I like especially the way, you can connect arbitrary jack applications together while they are running. This is a nice feature for the gmerlin-visualizer since it relies on grabbing the audio stream from another application.

Ambisonics rocks
I also heard about Ambisonics before but mostly ignored it. If you read the Wikipedia article, you get the impression, that ambisonics is a quite exotic recording/playback technique, which is used only by a small group of enthusiasts.

If you go to LAC, you get the impression that Ambisonics is the standard and things like stereo or 5.1 are quite exotic recording/playback techniques, which are used only by a small group of enthusiasts.

Audio vs. video development
I told the audio developers that they don't know how lucky they are. They agreed on an audio capture/playback architecture (jack), they all use the same internal format (floating point, not interleaved) for processing and they manage to make a conference each year.

In the video area, things are much worse. Capture and playback APIs exist, but they are far too many. Some people do internal processing in RGB, others prefer Y'CbCr. Converting to a common internal format is not feasible because video has much higher data rates than audio and conversion overhead must always be minimized. Also the video developer community is very fragmented, centered around different frameworks (each one being the standard). Cross project communication is sparse. The only common code is ffmpeg for video en-/decoding, but mostly because nobody wants to reimplement such a beast.

Gmerlin development
I talked a lot with Richard Spindler from the openmovieeditor, which makes heavy use of gmerlin libraries. We agreed that our collaboration works quite well because the development activities of gmerlin and openmovieeditor are nicely decoupled. Let's proceed like we already did.

Dirac video

2009-03-15T22:34:00.005+01:00

Libquicktime and gmerlin-avdecoder now support Dirac in quicktime. En- and decoding is done with the libschrödinger library. Having already implemented support for lots of other video codecs I noticed some things, both positive and negative.

Positive

Very precise specification of the uncompressed video format. Interlacing (including field order) is stored in the stream as well as singal ranges (video range or full range). This brings direct support for lots of colormodels.
Support for > 8 bit encoding. This is really a rare feature. While ffmpeg always sticks with 8 bit even for codecs with 10 bit or 12 bit modes, the libschrödinger API has higher precision options. Not sure if these modes are really supported internally by now.
It seems to aim for scalability from low-end internet downloads to intra-only modes for video editing applications. A lossless mode is also there. Whether it performs equally well for all usage scenarios is yet to be found out.
It pretends to be patent free. But since the patent jungle is so dense, it is almost impossible to prove this.
It has the BBC behind it, which hopefully means serious development, funding and a chance for a wide deployment.

Negative

Sequence end code in an own sample
In the quicktime mapping specification it is required, that the sequence end code (a 13 byte string telling that the stream ends here) must be in an own sample. This is a mess, since for all Quicktime codecs I know (even the most disgusting ones) 1 sample always corresponds to 1 frame. Having a sample, which is no frame, screws up the timing because there is a "duration" associated with the sequence-end-sample, which makes the total stream duration seem larger than is actually is. Also, a frame accurate demuxer will expect one frame more than the file actually has. For both libquicktime and gmerlin-avdecoder I wrote workarounds for this (they simply discard the last packet).

If I had written the mapping spec, I would require the sequence end code to be appended to the last frame (in the same sample). In addition it can be made optional since it's not really needed in quicktime.

If libquicktime encodes dirac files, everything is done according to the spec. Conformance to the spec is more important than my personal opinion about it :)

No ctts atom required
Quicktime timestamps (as given by the stts atom) are decoding timestamps. For all low-delay streams (i.e. streams without B-frames), these are equal to the presentation timestamps. For H.264 and MPEG4 ASP streams with B-frames, the ctts atom specifies the difference between PTS and DTS for each frame and lets the demuxer calculate correct presentation timestamps without touching the video data. If the ctts atom is missing, such quicktime files become as disgusting as AVIs with B-frames. Unfortunately the ctts atom isn't required by the mapping spec, which means we'll see such files in the wild.

The good news is, that the ctts atom isn't mentioned at all in the spec. From this I conclude that it is not forbidden either. Therefore, libquicktime always writes a ctts atom if the stream has B-frames.

On the decoding side (in gmerlin-avdecoder), the quicktime demuxer checks if a dirac stream has a ctts atom. If yes, it is used and everything is fine. If not (i.e. if the file wasn't written by libquicktime), a parser is fired up and an index must be built if one wants sample accuracy. The good news is that the parser is pretty simple and the same thing is needed anyway for decoding dirac in MPEG transport streams.

Other news

gmerlin-avdecoder also supports Dirac in Ogg and MPEG-2 transport streams.

Parallelizing video routines

2009-02-28T03:18:00.004+01:00

Basics
One project shortly after the last gmerlin release was to push the CPU usage beyond 50 % on my dual core machine. There is lots of talk about parallelization of commonly used video routines, but only little actual code.

In contrast to other areas (like scientific simulations) video routines are extremely simple to parallelize because most of them fulfill the following 2 conditions:

The outermost loop runs over scanlines of the destination image
The destination image is (except for a few local variables) the only memory area, where data are written

So the first step is to change the conversion functions of the type:

void conversion_func(void * data, int width, int height)
{
int i, j;
for(i = 0; i < height; i++)
  {
  for(j = 0; j < width; j++)
    {
    /* Do something here */
    }
  }
}

to something like:

void conversion_func(void * data, int width, int start, int end)
{
int i, j;
for(i = start; i < end; i++)
  {
  for(j = 0; j < width; j++)
    {
    /* Do something here */
    }
  }
}

The data argument points to a struct containing everything needed for the conversion like destination frame, source frame(s), lookup tables etc. The height argument was replaced by start and end arguments, which means that the function now processes a slice (i.e. a number of consecutive scanlines) instead of the whole image. The remaining thing now is to call the conversion function from multiple threads. It is important that the outermost loop is split to keep the overhead as small as possible.

The API perspective
Everything was implemented

With a minimal public API
Backwards compatible (i.e. if you ignore the API extensions things still work)
Completely independent of the actual thread implementation (i.e. no -lpthread is needed for gavl)

The whole magic can be boiled down to:

/* Slice process function, which is passed by gavl to the application */
typedef void (*gavl_video_process_func)(void * data, int start, int end);

/* Routine, which passes the actual work to the worker thread (which is managed by the application) */
typedef void (*gavl_video_run_func)(gavl_video_process_func func,
                               void * gavl_data,
                               int start, int end,
                               void * client_data, int thread);

/* Synchronization routine, which waits for the worker thread to finish */
typedef void (*gavl_video_stop_func)(void * client_data, int thread);

/* Set the maximum number of available worker threads (gavl might not need all of them
if you have more CPUs than scanlines) */
void gavl_video_options_set_num_threads(gavl_video_options_t * opt, int n);

/* Pass the run function to gavl */
gavl_video_options_set_run_func(gavl_video_options_t * opt,
                                gavl_video_run_func func,
                                void * client_data);

/* Pass the stop function to gavl */
void gavl_video_options_set_stop_func(gavl_video_options_t * opt,
                                 gavl_video_stop_func func,
                                 void * client_data);

Like always, the gavl_video_options_set_* routines have corresponding
gavl_video_options_get_* routines. These can be used for using the same
multithreading mechanism outside gavl (e.g. in a video filter).

The application perspective
As noted above there is no pthread specific code inside gavl. Everything is passed via callbacks. libgmerlin has a pthread based thread pool, which does exactly what gavl needs.

The thread pool is just an array of context structures for each thread:

typedef struct
{
/* Thread handle */
pthread_t t;

/* gavl -> thread: Do something */
sem_t run_sem;

/* thread -> gavl: I'm done */
sem_t done_sem;

/* Mechanism to make the fuction finish */
pthread_mutex_t stop_mutex;
int do_stop;

/* Passed from gavl */
void (*func)(void*, int, int);
void * data;
int start;
int end;
} thread_t;

The main loop (passed to pthread_create) for each worker thread looks like:

static void * thread_func(void * data)
{
thread_t * t = data;
int do_stop;
while(1)
  {
  sem_wait(&t->run_sem);

  pthread_mutex_lock(&t->stop_mutex);
  do_stop = t->do_stop;
  pthread_mutex_unlock(&t->stop_mutex);

  if(do_stop)
    break;
  t->func(t->data, t->start, t->len);
  sem_post(&t->done_sem);
  }
return NULL;
}

The worker threads are launched at program start and run all the time. As long as nothing is to do, they wait for the run_sem semaphore (using zero CPU). Launching new threads for every little piece of work would have a much higher overhead.

Passing work to a worker thread happens with the following gavl_video_run_func:

void bg_thread_pool_run(void (*func)(void*,int start, int len),
                     void * gavl_data,
                     int start, int end,
                     void * client_data, int thread)
{
bg_thread_pool_t * p     = client_data;
p->threads[thread].func  = func;
p->threads[thread].data  = gavl_data;
p->threads[thread].start = start;
p->threads[thread].end   = end;

sem_post(&p->threads[thread].run_sem);
}

Synchronizing (waiting for the work to finish) happens with the following gavl_video_stop_func:

void bg_thread_pool_stop(void * client_data, int thread)
{
bg_thread_pool_t * p     = client_data;
sem_wait(&p->threads[thread].done_sem);
}

The whole thread pool implementation can be found in the gmerlin source tree in lib/threadpool.c (128 lines including copyright header).

Implementations
Parallelization was implemented for

The video scaler
The colormatrix filters
The Image transformation
The Yadif port in the deinterlacing filter

Benchmarks
For benchmarking one needs a scenario where the parallelized processing routine(s) need the lions share of the total CPU usage. I decided to make a (completely nonsensical) gaussian blur with a radius of 50 over 1000 frames of a 720x576 PAL sequence. All code was compiled with default (i.e. optimized) options. Enabling 2 threads descreased the processing time from 81.31 sec to 43.35 sec.

Credits
Finally found this page for making source snippets in blogger suck less.

Elementary stream parsing

2009-02-27T00:54:00.004+01:00

See the image below for an updated block-schematics of gmerlin-avdecoder as addition to my last blog entry:

You see, what's new here: A parser, which converts "bad" packets into "good" packets. Now what does that exactly mean? A good (or well formed) packet has the following properties:

It always has a correct timestamp (the presentation time at least)
It has a flag to determine if the packet is a keyframe
It has a valid duration. This is necessary for frame accurate repositioning of the stream after seeking. At least if you want to support variable framerates.

Good packets (e.g. from quicktime files) can directly be passed to the decoder, and it's possible to build an index from them. Unfortunately some formats (most notably MPEG program- and transport streams), don't have such nice packets. You neither know where a frame starts, nor whether it is a keyframe. Large frames can be split across many packets, some packets can contain several small frames. Also timestamps are not available for each frame. To make things worse, these formats are very widely used. You find them in .mpg files, on DVDs, (S)VCDs, BluRay disks and in DVB broadcasts. Also all newer formats for consumer cameras (HDV and AVCHD) use MPEG-2 transport streams. Since they are important for video editing applications, so sample accurate access is a main goal here.

The first solutions for this were realized inside the decoders. libmpeg2 is very tolerant with regard to the frame alignment and ffmpeg has an AVParser, which splits a continuous stream into frames. Additional routines were written for building an index.

It was predictable that this would not be the ultimate solution. The decoders got very complicated and building the index was not possible without firing up an ffmpeg decoder (because the AVParser doesn't tell about keyframes) so index building was very slow.

So I spent some time to write a parsers for elementary A/V streams, which parse the streams to get all necessary information for creating well formed packets.

After that worked, I could be sure, that every codec always gets well-formed packets. What followed then, was by far the biggest cleanup in the history of gmerlin-avdecoder. Many things could be simplified, lots of cruft got kicked out, duplicate code got moved to central locations.

New features are:

Decoding of (and sample accurate seeking within) elementary H.264 streams
Sample accurate access for elementary MPEG-1/2 streams
Sample accurate access for MPEG program streams with DTS- and LPCM audio
Faster seeking because non-reference frames are skipped while approaching the new stream position
Much cleaner and simpler architecture.

The cleaner architecture doesn't necessarily mean less bugs, but they are easier to fix now :)

How gmerlin-avdecoder works

2008-12-11T20:11:00.005+01:00

If you study multimedia decoding software like xine, ffmpeg or MPlayer you find that they all work surprisingly similar. Gmerlin-avdecoder is no exception here. The important components are shown in the image below:

Input
The input module obtains the data. Examples for data sources are regular files, DVDs or DVB- or network streams. Usually the data is delivered in raw bytes. For DVDs and VCDs however, read and seek operations are sector based. Since both formats require that each sector must start with a syncpoint (an MPEG pack header) having sector based data in the demultiplexer speeds up several things.

Demultiplexer
This is, where the compressed frames are extracted from the container. During initialization, the demultiplexer creates the track table (bgav_track_table_t). This contains the tracks (in most cases just one). Each track contains the streams for audio, video and subtitles. For most containers the demultiplexer already knows the audio and video formats. For others, the codec must detect it. This means you should never trust the formats before you called bgav_start().

In some cases (DVD, VCD, DVB) the input already knows the complete track layout and which demultiplexer to use. The initialization of the demultiplexer can then skip the stream detection. In the general case the demultiplexer is selected according to the file content (i.e. the first few bytes). Some formats (MPEG, mp3) can have garbage before the first detection pattern, so we must repeatedly skip bytes before checking for one of these.

The demultiplexer has a routine, which reads the next packet from the input. Depending on the format, this involves decoding a packet header and extracting the compressed data, which can be handled by the codec later on. Some formats (rm, asf, MPEG-2 transport streams, Ogg) use 2-layer multiplexing. There are top-level packets which contain subpackets.

If the format is well designed, we also know the timestamps and duration of the packet and if the packet contains a keyframe. Not all formats are well designed though (or encoders are buggy), then we must do a lot of black magic to get as much information as possible.

Most demultiplexers are native implementations. As a fallback for very obscure formats we also support demultiplexing with libavformat.

Buffers
Gmerlin-avdecoder is strictly pull-based. If a video codec requests a packet, but the next packet in the stream belongs to an audio stream, it must be stored for later usage. For this, we have buffers, which are just chained lists of packets. They can grow dynamically. This approach makes the decoding process mostly insensitive to badly interleaved streams.

Interleaved vs. noninterleaved
The demultiplexing method described above reads the file strictly sequentially. This has the advantage that we never seek in the stream so we can do this for non-seekable sources.

Some files (more than you might think) are however completely non-interleaved, e.g. the audio packets follow all video packets. These always have a global index though. In this case, if a video packet is requested, the demultiplexer seeks to the packet start and reads the packet. This mode, which is also used in sample accurate mode, only works for seekable sources.

Codecs
These convert packets to A/V frames. In most cases, one packet equals one frame. In some cases (mostly for MPEG streams), the codec must first assemble frames from packets or split packets containing multiple frames. The codec outputs the gavl frames, which are handled by the application.

Codecs are selected according to fourccs. For formats, which don't have fourccs, we either invent them or we use fourccs from AVI or Quicktime.

Video codecs must care about timestamps. For MPEG streams the timestamps at multiplex level (with a 90 kHz clock) must be converted to the ones according to the video framerate. Audio codecs must do buffering because the the application can decide how many samples to read at once.

Text subtitles
There are no codecs for text subtitles. Each packet contains the string (converted to UTF-8 by the demultiplexer), the presentation timestamp and the duration.

Reading a Video frame
If an application calls bgav_read_video, the following happens:

The core calls the decode function of the codec
The codec checks if there is already a decoded frame available. This is the case after initialization because some codecs need to decode the first picture to detect the video format
If no frame is left, the codec decodes one. For this it will most likely need a new packet
The codec requests a packet. Either the packet buffer already has one, or the demultiplexer must get one
In streaming mode, the demultiplexer gets the next packet from the input and puts it into the packet buffer of the stream to which it belongs. If the stream is not used, no packet is produced and it's data is skipped. This is repeated until the end is reached or we found a packet for the video stream. If the end is reached, the demultiplexer signals EOF for the whole track.
In non-streaming mode the demultiplexer knows, which stream requested the packet. It seeks to the position of the next packet and reads it. EOF is signaled per stream.

Index building
Building file indices for sample accurate access can happen in different ways depending on the container format. In the end, we need byte positions in the file, the associated timestamps (in output timescale) and keyframe flags. The following modes are supported:

MPEG mode: The codec must build the index. This involves parsing the frames (only needed parts) to extract timing information with sample accuracy. Codecs supporting this mode are libmpeg2, libavcodec, libmad, liba52 and faad2.
Simple mode: The demultiplexer knows about the output timescale, gets precise timestamps and one packet equals one frame. Then no codec is needed for building the index.
Mix of the above. E.g. in flv timestamps are always in millseconds. This is precise for video streams. For audio streams (mostly mp3), we need MPEG mode.

B-frames are ommitted in the index. That's because noone will use them for a seekpoint anyway and the PTS get strictly monotone. This lets us do a fast binary search in the index, but the demultiplexer must be prepared for packets not contained in the index.

Downscaling algorithms

2008-12-05T22:27:00.014+01:00

The theory
Downscaling images is a commonly needed operation, e.g. if HD videos or photos from megapixel cameras are converterted for burning on DVD or uploading to the web. Mathematically, image scaling is exactly the same as audio samplerate conversion (only in 2D, but it can be decomposed into two subsequent 1D operations). All these have in common that samples of the destination must be calculated from the source samples, which are given on a (1D- or 2D-) grid with a different spacing (temporally or spatially). The reciprocal of the grid spacings are the sample frequencies.

In the general case (i.e. if the resample ratio is arbitrary) one will end up with interpolating the destination samples from the source samples. The interpolation method (nearest neighbor, linear, cubic...) can be configured in better applications.

One might think we are done here, but unfortunately we aren't. Interpolation is the second step of downscaling. Before that, we must make sure that the sampling theorem is not violated. The sampling theorem requires the original signal to be band-limited with a cutoff frequency of half the sample frequency. This cutoff frequency is also called the Nyquist frequency.

So if we upscale an image (or resample an audio signal to a higher sample frequency), we can assume that the original was already band-limited with half the lower lower sample frequency so we have no problem. If we downsample, we must first apply a digital low-pass filter to the image. Low-pass filtering of images is the same as blurring.

The imagemagick solution
What pointed me to this subject in the first place was this post in the gmerlin-help forum (I knew about sampling theory before, I simply forgot about it when programming the scaler). The suggestion, which is implemented in ImageMagick, was to simply widen the filter kernel by the inverse scaling ratio. For linear downscaling to 1/2 size this would mean to do an interpolation involving 4 source pixels instead of 2. The implementation extremely simple, it's just one additional variable for calculating the number and values of filter coefficients. This blurs the image because it does some kind of averaging involving more source pixels. Also the amount of blurring is proportional to the inverse scale factor, which is correct. The results actually look ok.

The gavl solution
I thought what's the correct way to do this. As explained above, we must blur the image with a defined cutoff frequency and then interpolate it. For the blur filter, I decided to use a Gaussian low-pass because it seems the best suited for this.

The naive implementation is to blur the source image into a temporary video frame and then interpolate into the destination frame. It can however be done much faster and without temporary frame, because the 2 operations are both convolutions. And the convolution has the nice property, that it's associative. This means, that we can convolve the blur coefficients with the interpolation coefficients resulting in the filter coefficients for the combined operation. These are then used on the images.

The difference
The 2 solutions have a lot in common. Both run in 1 pass and blur the image according to the inverse scaling ratio. The difference is, that the imagemagick method simply widens the filter kernel by a factor while gavl widens the filter by colvolving with a low-pass.

Examples
During my research I found this page. I downloaded the sample image (1000x1000) and downscaled it to 200x200 with different methods.

First the scary ones:

OpenGL (Scale mode linear, GeForce 8500 GT, binary drivers)

XVideo (GeForce 8500 GT, binary drivers)

Firefox 3.0.4 (that's why I never let the browser scale when writing html)

Gimp (linear, indexed mode)

gavl (downscale filter: GAVL_DOWNSCALE_FILTER_NONE)

Now it gets better:

Gimp (linear, grayscale mode)

mplayer -vf scale=200:200 scale.mov
The movie was made with qtrechunk from the png file.

gavl with imagemagick method (downscale filter: GAVL_DOWNSCALE_FILTER_WIDE)

gavl with gaussian preblur (downscale filter: GAVL_DOWNSCALE_FILTER_GAUSS)

Blogger thumbnail (400x400). Couldn't resist to upload the original size image to blogger and see what happens. Not bad, but not 200x200.

Video4vimeo

2008-11-29T20:58:00.024+01:00

Everyone uploads videos nowadays. Specialists use vimeo because youtube quality sucks. So the project goal was to create a video file for upload to Vimeo with Gmerlin-transcoder, optimize the whole process and fix all bugs.

The footage

I had to make sure that I own the copyright of the example clip and that nobodies privacy is violated. So I decided to make a short video featuring a toilet toy I bought in Tokyo in 2003. A friend of mine went to the shop a few years later, but it was already sold out.

The equipment

My camera is a simple mini-DV one. It's only SD but since the gmerlin architecture is nicely scalable, the same encoder settings (except the picture size) should apply for HD as well. I connected the camera via firewire and recorded directly to the PC (no tape involved) with Kino.

Capture format

The camera sends DV frames (with encapsulated audio) via firewire to the PC. This format is called raw DV (extension .dv). The Kino user can choose whether to wrap the DV frames into AVI or Quicktime or export them raw. Since the raw DV format is completely self-contained, it was choosen as input format for Gmerlin-transcoder. Wrapping DV into another container makes only sense for toolchains, which cannot handle raw DV.

Quality considerations

My theory is that the crappy quality of many web-video services is partly due to financial considerations of the service providers (crappy files need less space on the server and less bandwidth for transmission), but partly also due to people making mistakes when preparing their videos. Here are some things, which should be kept in mind:

1. You never do the final compression
In forums you often see people asking: How can I convert to flv for upload on youtube? The answer is: Don't do it. Even if you do it, it's unlikely that the server will take your video as it is. Many video services are known to use ffmpeg for importing the uploaded files, which can read much more than just flv. Install ffmpeg to check if it can read your files.

Compression parameters should be optimized for invisible artifacts in the file you upload. That's because in the final compression (out of your control) will add more artifacts. And 2nd generation artifacts look even more ugly, the results can be seen on many places in the web.

2. Minimize additional conversions on the server
If you scale your video to the same size it will have on the server, chances are good that the server won't rescale it. The advantage is that scaling will happen for the raw material, resulting in minimal quality loss. Scaled video looks ugly if the original has compression artifacts, which would be the case if you let the server scale.

3. Don't forget to deinterlace
Interlaced video compressed in progressive mode looks extraordinarily ugly. Even more disappointing is that many people apparently forget to deinterlace. Even the crappiest deinterlacer is better than nothing.

4. Minimize artifacts by prefiltering
If, for whatever reason, artifacts are unavoidable you can minimize them by doing a slight blurring of the source material. Usually this shouldn't be necessary.

Format conversion

All video format conversions can be done in a single pass by the Crop & Scale filter. This gives maximum speed, smallest rounding errors and smallest blurring.

Deinterlacing
Sophisticated deinterlacing algorithms are only meaningful if the vertical resolution should be preserved. In our case, where the image is scaled down anyway, it's better to let the scaler deinterlace. Doing scaling and deinterlacing in one step also decreases the overall blurring of the image.

Scaling
Image size for Vimeo in SD seems to be 504x380. It's the size of their flash widget and also the size of the .flv video. Square pixels are assumed.

Cropping
The aspect ratio of PAL DV is a bit larger than 4:3. Also 504x380 with square pixels is not exactly 4:3. Experiments have shown, cropping by 10 pixels each on the left and right borders removed black border at the top and bottom. If your source material has a different size, these values will be different as well.

Chroma placement
Chroma placement for PAL DV is different from H.264 (which has the same chroma placement as MPEG-2). Depending on the gavl quality settings, this fact is either ignored or a another video scaler is used for shifting the chroma locations later on. I thought that could be done smarter.

Since the gavl video scaler can do many things at the same time (it already does deinterlacing, cropping and scaling) it can also do chroma placement correction. For this, I made the chroma output format of the Crop & scale filter configurable. If you set this to the format of the final output, subsequent scaling operations are avoided.

Since ffmpeg doesn't care about chroma placement it's probably unnecessary that we do. On the other hand, our method has zero overhead and does practically no harm.

Audio
Vimeo wants audio to be sampled at 44,1 kHz, most cameras record in 48 kHz. The following settings take care for that:

Encoding

The codecs are H.264 for video and AAC for audio. Not only because they are recommended by vimeo, they give indeed the best results for a given bitrate.

For some reason, vimeo doesn't accept the AAC streams in Quicktime files created by libquicktime. Apple Quicktime, mplayer and ffmpeg accept them and I found lots of forum posts describing exactly the same problem. So I believe that this is a vimeo problem.

The solution I found is simple: Use mp4 instead of mov. People think mp4 and mov are indentical, but that's not true. At least in this case it makes a difference. The compressed streams are, however, the same for both formats.

Format

The make streamable option is probably unnecessary, but I allow people to download the original .mp4 file and maybe they want to watch it while downloading.

Audio codec

The default quality is 100, I increased that to 200. Hopefully this isn't the reason vimeo rejects the audio when in mov. The Object type should be Low (low complexity). Some decoders cannot decode anything else.

Video codec

I decreased the maximum GOP size to 30 as recommended by Vimeo. B-frames still screw up some decoders, so I didn't enable them. All other settings are default.

I encode with constant quality. In quicktime, there is no difference between CBR and VBR video, so the decoder won't notice. Constant quality also has the advantage that this setting is independent from the image size. The quantizer parameter was decreased from 26 to 16 to increase quality. It could be decreased further.

Bugs

The following bugs were fixed during that process:

Reading raw DV files was completely broken. I broke it when I implemented DVCPROHD support last summer.
Chroma placement for H.264 is the same as for MPEG-2. This is now handled correctly by libquicktime and gmerlin-avdecoder.
Blending of text subtitles onto video frames in the transcoder was broken as well. It's needed for the advertisement banner at the end.
Gmerlin-avdecoder always signalled the existance of timecodes for raw DV. This is ok if the footage comes from a tape, but when recording on the fly my camera produces no timecodes. This resulted in a Quicktime file with a timecode track, but without timecodes. Gmerlin-avdecoder was modified to tell about timecodes only if the first frame actually contains a timecode.
For making the screenshots, I called
LANG="C" gmerlin_transcoder
This switched the GUI to English, except the items belonging to libquicktime. I found, that libquicktime translated the strings way to early (the German strings were saved gmerlin plugin registry). I made a change to libquicktime so that the strings are only translated for the GUI widget. Internally they are always English.

The result

Make your webcam suck less

2008-11-19T23:49:00.009+01:00

Every webcam sucks. Not because of the webcam itself, but because of the way it's handled by the software. Some programs support only RGB formats, others work only in YUV. Supporting all pixelformats directly by the hardware would increase the price of these low-end articles. Supporting all pixelformats by the drivers would mean to have something similar to gavl in the Linux kernel. The linux kernel developers don't want this because it belongs into userspace. They are right IMO. And since not all programs have proper pixelformat support, you can always find an application, which doesn't support your cam.

Other problems are that some webcams flip the image horizontally (for reasons I don't want to research). Furthermore, some programs aren't really smart when detecting webcams. They stop at the first device (which can be TV-card instead of a webcam) they can't handle.

So the project was to make a webcam device at /dev/video0, which supports as many pixelformats as possible and allows image manipulation (like horizontal flipping).

The solution involved the following:

Wrote a V4L2 input module for the real webcam (not directly necessary for this project though).
Fixed my old webcam tool camelot. Incredible how software breaks, if you don't maintain it for some time.
Added support for gmerlin filters in camelot: These can not only correct the image flipping, they provide tons of manipulation options. Serious ones and funny ones.
Added an output module for vloopback. It's built into camelot and provides the webcam stream through a video4linux (1, not 2) device. It supports most video4linux pixelformats because it has the conversion power of gavl behind it. Vloopback is not in the standard kernel. I got it from svn with

svn co http://www.lavrsen.dk/svn/vloopback/trunk/ vloopback

A tiny initialization script (to be called as root) initializes the kernel modules:

#!/bin/sh
# Remove modules if they were already loaded
rmmod pwc
rmmod vloopback

# Load the pwc module, real cam will be /dev/video3
modprobe pwc dev_hint=3

# Load the vloopback module, makes /dev/video1 and /dev/video2
modprobe vloopback dev_offset=1

# Link /dev/video2 to /dev/video0 so even stupid programs find it
ln -sf /dev/video2 /dev/video0

Instead of the pwc module, you must the one appropriate for your webcam. Not sure if all webcam drivers support the dev_hint option.

My new webcam works with the following applications:

These are all I need for now. Not working are kopete, flash and Xawtv.

Gmerlin pipelines explained

2008-11-13T20:36:00.005+01:00

Building multimedia software on top of gavl saves a lot of time others spend on writing optimized conversion routines (gavl already has more than 2000 of them) and bullet-proof housekeeping functions.

On the other hand, gavl is a low-level library, which leaves lots of architectural decisions to the application level. And this means, that gavl will not provide you with fully featured A/V pipelines. Instead, you have to write them yourself (or use libgmerlin and take a look at include/gmerlin/filters.h and include/gmerlin/converters.h).

I'm not claiming to have found the perfect solution for the gmerlin player and transcoder, but nevertheless here is how it works:

Building blocks
The pipelines are composed of

A source plugin, which gets A/V frames from a media file, URL or a hardware device
Zero or more filters, which somehow change the A/V frames
A destination plugin. In the player it displays video or sends audio to the soundcard. For the transcoder, it encodes into media files.
Format converters: These are inserted on demand between any two of the above elements

Asynchronous pull approach
The whole pipeline is pull-based. Pull-based means, that each component requests data from the preceeding component. Asynchronous means that (in contrast to plain gavl), we make no assumption on how many frames/samples a component needs at the input for producing one output frame/sample. This makes it possible to do things like framerate conversion or framerate-doubling deinterlacing. As a consequence, filters and converters which remember previous frames need a reset function to forget about them (the player e.g. calls them after seeking).

Unified callbacks
In modular applications it's always important that modules know as little as possible about each other. For A/V pipelines this means, that each component gets data from the preceeding component using a unified callback, no matter if it's a filter, converter or source. There are prototypes in gmerlin/plugin.h

typedef int (*bg_read_audio_func_t)(void * priv, gavl_audio_frame_t* frame, int stream,
                                 int num_samples);

typedef int (*bg_read_video_func_t)(void * priv, gavl_video_frame_t* frame, int stream);

These are provided by input plugins, converters and filters. The stream argument is only meaningful for media files which have more than one audio or video stream. How the pipeline is exactly constructed (e.g. if intermediate converters are needed) matters only during initialization, not in the time critical processing loop.

Asynchronous vs synchronous
As noted above, some filter types are only realizable if the architecture is asynchronous. Another advantage is that for a filter, the input- and output frame can be the same (in-place conversion). E.g. the timecode tweak filter of gmerlin looks like:

typedef struct
{
bg_read_video_func_t read_func;
void * read_data;
int read_stream;

/* Other stuff */
/* ... */
} tc_priv_t;

static int read_video_tctweak(void * priv, gavl_video_frame_t * frame,
                      int stream)
{
tc_priv_t * vp;
vp = (tc_priv_t *)priv;

/* Let the preceeding element fill the frame, return 0 on EOF */
if(!vp->read_func(vp->read_data, frame, vp->read_stream))
 return 0;

/* Change frame->timecode */
/* ... */

/* Return success */
return 1;
}

A one-in-one-out API would need to memcpy the video data only for changing the timecode.

Of course in some situations outside the scope of gmerlin, asynchronous pipelines can cause problems. This is especially the case in editing applications, where frames might be processed out of order (e.g. when playing backwards). How to solve backwards playback for filters, which use previous frames, is left to the NLE developers. But it would make sense to mark gmerlin filters, which behave synchronously (most of them actually do), as such so we know we can always use them.

Release plans

2008-11-09T15:13:00.004+01:00

Time to make releases of the packages. The current status is the following:

Gmerlin-mozilla is practically ready, some well hidden bugs cause it to crash sometimes though. Also the scripting interface could be further extended.
gavl is ready. New features since the last version are timecode support, image transformation and a contributed varispeed capable audio resampler.
gmerlin-avdecoder got lots of fixes, support for newer ffmpegs, a demuxer for redcode files and RTP/RTSP support. The latter was most difficult to implement. It still needs some work for better recovering after packet loss in UDP mode. Since all important features are implemented now, gmerlin-avdecoder will get the version 1.0.0.
The GUI player can now import directories with the option "watch directory". This will cause the album to be syncronized with the directory each time it is opened. The plan is to further extend this such that even an opened album is regularly synchronized via inotify. Except from this, the gmerlin package is ready.

Introducing gmerlin-mozilla

2008-11-06T00:39:00.008+01:00

What's the best method to check if your multimedia architecture is really completely generic and reusable? One way is to write a firefox plugin for video playback and beat onto everything until it no longer crashes. The preliminary result is here:

And here are some things I think are worth noting:

How a plugin gets it's data
There are 2 methods:

firefox handles the TCP connection and passes data via callbacks (NPP_WriteReady, NPP_Write). The gmerlin plugin API got a callback based read interface for this.
the plugin opens the URL itself

The first method has the advantage, that procotols not supported by gmerlin but by firefox (e.g. https) will work. The disadvantage is, that passing the data from firefox to the input thread if the player will almost lockup the GUI because firefox spends most of it's time waiting until the player can accept more data. I found no way to prevent this in an elegant way. Thus, such streams are written to a temporary file and read by the input thread. Local files are recognized as such and opened by the plugin.

Emulating other plugins
Commercial plugins (like Realplayer or Quicktime) have lots of gimmicks. One of these lets you embed multiple instances of the plugin, where one will show up as the video window, another one as the volume slider etc. Gmerlin-mozilla handles these pragmatically: The video window always has a toolbar (which can hide after the mouse was idle), so additional control widgets are not initialized at all. They will appear as grey boxes.

Of course not all oddities are handled correctly yet, but the infrastructure for doing this is there.

Scriptability
Another gimmick is to control the plugin from a JavaScript GUI. While the older scripting API (XPCOM) was kind of bloated and forced the programmer into C++, the new method (npruntime) is practically as versatile, but much easier to support (even in plain C). Basically, the plugin exports an object, (an NPObject) which has (among others) functions for querying the supported methods and properties. Other functions exist for invoking methods, or setting and getting properties. Of course not all scripting commands are supported yet.

GUI
A GUI for a web-plugin must look sexy, that's clear. Gmerlin-mozilla has a GUI similar to the GUI player (which might look completely unsexy for some). But in contrast to other free web-plugins it's skinnable, so there is at least a chance to change this.

Some GUI widgets had to be updated and fixed before they could be used in the plugin. Most importantly, timeouts (like for the scrolltext) have to be removed from the event-loop before the plugin is destroyed, otherwise a crash happens after.

The fine thing is, that firefox also uses gtk-2 for it's GUI, so having Gtk-widgets works perfectly. If the browser isn't gtk-2 based, the plugin won't load.

Embedding technique
Gmerlin-mozilla needs XEmbed. Some people hate XEmbed, but I think it's pretty well designed as long as you don't expect too much from it. The Gmerlin X11 display plugin already supports XEmbed because it always opens it's own X11 connection. After I fixed some things, it embeds nicely into firefox.

Configuration
The GUI should not be bloated by exotic buttons, which are rarely used. Therefore most of the configuration options are available via the right-click menu. Here, you can also select fullscreen mode.

Streaming through the NAT

2008-10-16T00:43:00.012+02:00

The only missing network streaming protocol for gmerlin-avdecoder was RTSP/RTP, so I decided to implement it. Some parts (the ones needed for playing the Real-rtsp variant) were already there, but the whole RTP stuff was missing.

The advantage of these is that they are well documented in RFCs. With some knowledge about how sockets and their API work, implementation was straightforward. Special about RTSP/RTP is, that there is one RTSP connection (usually TCP port 554) which acts like a remote control, while the actual A/V data are delivered over RTP, which usually uses UDP. To make things more complicated, each stream is transported over an own UDP socket, with another socket used for Qos infos. Playing a normal movie with audio and video needs 4 UDP sockets then.

The basic functions were implemented, I opened 4 UDP ports on my DSL-Router and I could play movies :)

Then I stumbled across something strange:

My code worked completely predictable regarding the router configuration. When I closed the ports on the router (or changed the ports in my code), it stopped working
Both ffmpeg and vlc (which, like MPlayer, uses live555 for RTSP) always work in UDP mode, no need to manually open the UDP ports. Somehow they make my router forward the incoming RTP packets to my machine.

So the question was: How?

After spending some time with wireshark and strace I made sure, that I setup my sockets the same way as the other applications, and the RTSP requests are the same. When gmerlin-avdecoder still didn't make it through the NAT (and me almost freaking out) I decided to take a look at some TCP packets, which were marked with the string "TCP segment of a reassembled PDU". I noticed that these occur only in the wireshark dump of gmerlin-avdecoder, not in the others.

After googling a bit, the mystery was solved:

The Router (which was found to be a MIPS-based Linux box) recognizes the RTSP protocol. By parsing the client_port field of the SETUP request it knows which UDP ports it must open and forward to the client machine.
The "TCP segment of a reassembled PDU" packets are small pieces belonging to one larger RTSP request.
If the SETUP line is not in the same TCP packet as the line which defines the transport, the recognition by the router will fail.
Wireshark fooled me by assembling the packets belonging to the same request into a larger one and displaying it together with the pieces (this feature can be turned off in the wireshark TCP configuration).
The fix was simple: I write the whole request into one string, and send this string at once. Finally the router automagically sends the RTP packets to gmerlin-avdecoder.

What did I learn through this process? Most notably that TCP is stream-based only for the client and the server. Any hardware between these 2 only sees packets. Applications relying on intelligent network hardware must indeed take care, which data will end up in which packet.

You might think that it's actually no problem to open UDP ports on the router, and doing such things manually is better than automatically. But then you'll have many people running their clients with the same UDP ports, which makes attacks easier for the case that gmerlin-avdecoder has a security hole. Much better is to choose the ports randomly. Then, we can also have multiple clients in the same machine. The live555 library uses random ports, ffmpeg doesn't.

Globals in libs: If and how

2008-10-08T23:39:00.002+02:00

As a follow-up to this post I want to concentrate on the cases, where global variables are tolerable and how this should be done.

Tolerable as global variables are data, whose initialization must be done at runtime and
takes a significant amount of time. One example is the libquicktime codec registry. It's creation involves scanning the plugin directory, comparing the contents with a registry and loading all modules (with a time consuming dlopen), for which the registry entries are missing or outdated. This is certainly not something, which should be done per instance (i.e. for each opened file). Other libraries have similar things.

Next question is how can they be implemented? A simple goal is, that the library must linkable with a plugin (i.e. dynamic module) instead of an executable. This means, that repeated loading and unloading (from different threads) must work without any problems. A well designed plugin architecture knows as little as possible about the plugins, so having global reference counters for each library a plugin might link in, is not possible.

Global initialization and cleanup functions

Many libraries have functions like libfoo_init() and libfoo_cleanup(), which are to be called before the first and after the last use of other functions from libfoo respectively. This causes problems for a plugin, which has no idea if this library was already be loaded/initialized by another plugin (or by another instance of itself). Also before a plugin is unloaded there is no way to find out, if libfoo_cleanup() can safely be called or if this will crash another plugin. Omitting the libfoo_cleanup() call opens a memory leak if the libfoo_init() function allocated memory. From this we find that the global housekeeping functions are ok if either:

Initialization doesn't allocate any resources (i.e the cleanup function is either a noop or missing) and
Initialization is (thread safely) protected against multiple calls

or:

Initialization and cleanup functions maintain an internal (thread safe) reference counter, so that only the first init and last cleanup call will actually do something

Initialization on demand, cleanup automatically

This is how the libquicktime codec registry is handled. It meets the above goals but doesn't need any global functions. Initialization on demand means, that the codec registry is initialized before it's accessed the first time. Each function, which accesses the registry starts with a call to lqt_registry_init(). The subsequent registry access is enclosed by lqt_registry_lock() and lqt_registry_unlock(). These 3 functions do the whole magic and they look like:


static int registry_init_done = 0;
pthread_mutex_t codecs_mutex = PTHREAD_MUTEX_INITIALIZER;

void lqt_registry_lock()
  {
  pthread_mutex_lock(&codecs_mutex);
  }

void lqt_registry_unlock()
  {
  pthread_mutex_unlock(&codecs_mutex);
  }

void lqt_registry_init()
  {
  /* Variable declarations omitted */
  /* ... */

  lqt_registry_lock();
  if(registry_init_done)
    {
    lqt_registry_unlock();
    return;
    }

  registry_init_done = 1;

  /* Lots of stuff */
  /* ... */

  lqt_registry_unlock();
  }

We see that protection against multiple calls is garantueed. The protection mutex itself initialized from the very beginning (before the main function is called).

While this initialization should work on all POSIX systems, automatic freeing is a bit more tricky and only possible for gcc (don't know if other compilers have similar features). The best time for freeing global resources is right before the library is unloaded. Most binary formats let you mark functions, which should be called before unmapping the library (in ELF files, this is done by putting these into the .fini section). In the sourcecode, this looks like:


#if defined(__GNUC__)

static void __lqt_cleanup_codecinfo() __attribute__ ((destructor));

static void __lqt_cleanup_codecinfo()
  {
  lqt_registry_destroy();
  }

#endif

Fortunately the dlopen() and dlclose() functions maintain reference counts for each module. So the cleanup function is garantueed to be called by the dlclose() call, which unloads the last instance of the last plugin linked to libquicktime.

I regularly check my programs for memory leaks with valgrind. Usually (i.e. after I fixed my own code) all remaining leaks come from libraries, which miss some of the goals described above.

Remove globals from libs

2008-10-08T21:41:00.005+02:00

Global variables in libraries are bad, everyone knows that. Maybe I'll make another post explaining when they can be tolerated how this can be done. But for now we assume that they are simply bad.

One common mistake is to declare static data (like tables) as non-const. This makes them
practically variables. If the code never changes them, they cause no problem in terms of thread safety. But unfortunately the dynamic linker doesn't know that, so they will be mapped into r/w pages when the library is loaded. And those pages will, of course, not be shared between applications so you end up with predictable redundant blocks in your precious RAM.

Cleaning this up is simple: Add const to all declarations, where it's missing. But how does one find all these declarations in a larger sourcetree in a reasonable time? The ELF format is well documented and there are numerous tools to examine ELF files.

Let's take the following C file and pretend it's a library build from 100s of sourcefiles with 100000 of codelines:


struct s
{
char * str;
char ** str_list;
int i;
};

struct s static_data_1 =
{
"String1",
(char*[]){ "Str1", "Str2" },
1,
};

char * static_string_1 = "String2";

int zeroinit = 0;

Now there are 2 sections in an ELF file, which need special attention: The .data section contains statically initialized variables. The .bss section contains data which is initialized to zero. After compiling the file with gcc -c the sizes of the sections can be obtained with:

# size --format=SysV global.o
global.o  :
section           size   addr
.text                0      0
.data               56      0
.bss                 4      0
.rodata             26      0
.comment            42      0
.note.GNU-stack      0      0
Total              128

So we have 56 bytes in .data and 4 bytes in .bss. After successful cleanup all these should ideally end up in the .rodata section (read-only statically initialized data). Since we have 100000 lines of code, the next step is to find the variable names (linker symbols) contained in the sections:

# objdump -t global.o

global.o:     file format elf64-x86-64

SYMBOL TABLE:
0000000000000000 l    df *ABS* 0000000000000000 global.c
0000000000000000 l    d  .text 0000000000000000 .text
0000000000000000 l    d  .data 0000000000000000 .data
0000000000000000 l    d  .bss 0000000000000000 .bss
0000000000000000 l    d  .rodata 0000000000000000 .rodata
0000000000000020 l     O .data 0000000000000010 __compound_literal.0
0000000000000000 l    d  .note.GNU-stack 0000000000000000 .note.GNU-stack
0000000000000000 l    d  .comment 0000000000000000 .comment
0000000000000000 g     O .data 0000000000000018 static_data_1
0000000000000030 g     O .data 0000000000000008 static_string_1
0000000000000000 g     O .bss 0000000000000004 zeroinit

Now we know that the variables static_data_1, static_string_1
and zeroinit are affected.

The symbol __compound_literal.0 comes from the expression (char*[]){ "Str1", "Str2" }. The bad news is that compound literals are lvalues according to the C99 standard, so they won't be assumed const by gcc. You can declare them const, but they'll still be in the .data section, at least for gcc-Version 4.2.3 (Ubuntu 4.2.3-2ubuntu7). The cleaned up file looks like:

struct s
{
const char * str;
char ** const str_list;
int i;
};

static const struct s static_data_1 =
{
"String1",
(char*[]){ "Str1", "Str2" },
1,
};

char const * const static_string_1 = "String2";

const int zeroinit = 0;

The resulting symbol table:

0000000000000000 l    df *ABS* 0000000000000000 global1.c
0000000000000000 l    d  .text 0000000000000000 .text
0000000000000000 l    d  .data 0000000000000000 .data
0000000000000000 l    d  .bss 0000000000000000 .bss
0000000000000000 l    d  .rodata 0000000000000000 .rodata
0000000000000010 l     O .rodata 0000000000000018 static_data_1
0000000000000000 l     O .data 0000000000000010 __compound_literal.0
0000000000000000 l    d  .note.GNU-stack 0000000000000000 .note.GNU-stack
0000000000000000 l    d  .comment 0000000000000000 .comment
0000000000000040 g     O .rodata 0000000000000008 static_string_1
0000000000000048 g     O .rodata 0000000000000004 zeroinit

Larger libraries have huge symbol tables, so you will of course filter it with:

grep \\.data | grep -v __compound_literal

So if you want to contribute to a library which needs some cleanup, and you are of the "I know just a little C but I want to help"-type, this is a good idea for a patch :)

2 ssh servers on the same port

2008-10-08T00:37:00.002+02:00

A tcp port can only be used by one server process for incoming connections. If another process wants to listen on the same port it will get an "address already in use" error from the OS. If you know the background it's pretty clear why it must be so.

But imagine a case like the following:

You want to make a linux machine reachable via ssh

From the same subnet passwords are sufficient

From outside only public key authentication is allowed

Your users are already happy if they get their ssh clients working on Windows XP. You don't want to bother them (and indirectly yourself as the admin) with nonstandard port numbers.

Your sshd doesn't support different configurations depending on the source address.

At a first glance, this looks unsolvable. But if you have an iptables firewall (and you will have one if the machine is worldwide reachable) there is a little known trick called port redirection.

You run 2 ssh servers: The external one (with public key authentication) listens at port 22, the internal one (with passwords) listens e.g. at port 2222. Then you configure your iptables such, that incoming packets which come from the subnet to port 22 are redirected to port 2222. The corresponding lines in the firewall script look like:


# Our Subnet
SUB_NET="192.168.1.0/24"

# iptables command
IPTABLES=/usr/sbin/iptables

# default policies, flush all tables etc....
...

# ssh from our subnet (redirect to port 2222 and let them through)
$IPTABLES -t nat -A PREROUTING -s $SUB_NET -p tcp --dport 22 \
 -j REDIRECT --to-ports 2222
$IPTABLES -A INPUT -p tcp -s $SUB_NET --syn --dport 2222 -j ACCEPT

# ssh from outside
$IPTABLES -A INPUT -p tcp -s ! $SUB_NET --syn --dport 22 -j ACCEPT

I have this configuration on 2 machines for many months now with zero complaints so far.

The fastest loop in C

2008-09-24T23:36:00.005+02:00

Sure, practically nobody knows assembler nowadays. But if you use a certain construct very often in time critical code, you might get curious what's the fastest way to do it. One example are loops. I tested this with a loop of the following properties:

Loop count is unknown at compile time, so the loop cannot be unrolled
The loop index i is not used within the loop body. In particular it doesn't matter it it's incremented or decremented

This kind of loop occurs very often in gavl colorspace conversion routines. The following C-file has 3 functions, which do exactly the same:


void do_something();

static int num;

void loop_1()
{
int i;
for(i = 0; i < num; i++)
  do_something();
}

void loop_2()
{
int i = num;
while(i--)
  do_something();
}

void loop_3()
{
int i = num+1;
while(--i)
  do_something();
}

This code can be compiled with gcc -O2 -S. The resulting loop bodies are the following:



loop 1:

.L17:
 xorl    %eax, %eax
 addl    $1, %ebx
 call    do_something
 cmpl    %ebp, %ebx
 jne     .L17

loop 2:

.L11:
 xorl    %eax, %eax
 addl    $1, %ebx
 call    do_something
 cmpl    %ebp, %ebx
 jne     .L11

loop 3:

.L5:
 xorl    %eax, %eax
 call    do_something
 subl    $1, %ebx
 jne     .L5

As you see, the winner is loop 3. Here, the whole logic (without initialization) needs 2 machine instructions:

Decrement ebx
Jump is result is nonzero

I doubt it can be done faster. You also see, that loop 2 looks simpler than loop 1 in C, but the compiler produces exactly the same code. I changed the innermost gavl pixelformat conversion loops to loop 3 and could even measure a tiny speedup for very simple conversions.

Fun with the colormatrix

2008-09-22T20:52:00.005+02:00

Some years ago, when I wrote lemuria, I was always complaining that OpenGL doesn't have a colormatrix. Later I found, that some extensions provide this, but not the core API. Today I think the main problem was that I used a graphics system (OpenGL) which is optimized to look as realistic as possible for a visualization which should look a surrealistic as possible :)

Later when I concentrated on more serious work like developing high quality video filters, I rediscovered the colormatrix formalism. Now what is that all about and what's so exciting about that? If you assume each pixel to be a vector of color channels, multiplying the vector by a matrix is simply a linear transformation of each pixel. In homogenous coordinates and in RGBA or YUVA (i.e. 4 channel) colorspace, it can be completely described by a 4x5 matrix. Now there are lots of commonly used video filters, which can be described by a colormatrix multiplication:

Brightness/Contrast (Y'CbCrA)

[	c	0	0	0	b - (1+c)/2	]
[	0	1	0	0	0	]
[	0	0	1	0	0	]
[	0	0	0	1	0	]

b and c are between 0.0 and 2.0.

Saturation/Hue rotation (Y'CbCrA)

[	1	0	0	0	]
[	0	s*cos(h)	-s*sin(h)	0	]
[	0	s*sin(h)	s*cos(h)	0	]
[	0	0	0	1	]

h is between -pi and pi (0 is neutral). s is between 0.0 (grayscale) and 2.0 (oversaturated).

Invert single RGB channels (RGBA)

[	-1	0	0	0	1	]
[	0	1	0	0	0	]
[	0	0	1	0	0	]
[	0	0	0	1	0	]

This inverts only the first (red) channel. Inverting other channels is done by changing any of the other lines accordingly.

Swap RGB channels (RGBA)

[	0	0	1	0	]
[	0	1	0	0	]
[	1	0	0	0	]
[	0	0	0	1	]

This swaps the first (red) and 3rd (blue) channel. It can be used to rescue things if some buggy routine confused RGB and BGR. Other swapping schemes are trivial to do.

RGB gain (RGBA)

[	g_r	0	0	0	]
[	0	g_g	0	0	]
[	0	0	g_b	0	]
[	0	0	0	1	]

g_r, g_g and g_b are in the range 0.0..2.0.

There are of course countless other filters possible, like generating an alpha channel from (inverted) luminance values etc. Now, what do you do if you want to make e.g. a brightness/contrast filter working on RGBA images? The naive method is to transform the RGBA values to Y'CbCrA, do the filtering and transform back to RGBA. And this is what can be optimized in a very elegant way by using some simple matrix arithmetics. Instead of performing 2 colorspace conversions in addition to the actual filter for each pixel, you can simply transform the colormatrix (M) from Y'CbCrA to RGBA:

Untransformed pixel in RGBA:	(p)
Untransformed pixel in Y'CbCrA:	(RGBA->Y'CbCrA)*(p)
Transformed pixel in Y'CbCrA:	(M)(RGBA->Y'CbCrA)(p)
Transformed pixel in RGBA:	(Y'CbCrA->RGBA)(M)(RGBA->Y'CbCrA)*(p)

The matrices (RGBA->Y'CbCrA) and (Y'CbCrA->RGBA) are the ones which convert between RGBA and Y'CbCrA. Since matrix multiplication is associative you can combine the 3 matrices to a single one during initialization:

(M)_RGBA = (Y'CbCrA->RGBA)*(M)_Y'CbCrA*(RGBA->Y'CbCrA)

If you have generic number-crunching routines, which to the vector-matrix multiplication for all supported pixelformats, you can reduce conversion overhead in your processing pipeline significantly.

Another trick is to combine multiple transformations in one matrix. The gmerlin video equalizer filter does this for brightness/contrast/saturation/hue rotation.

A number of gmerlin filters use the colormatrix method. Of course, for native pixelformats (image colorspace = matrix colorspace), the transformation is done directly, since it usually needs much less operations than a generic matrix-vector multiplication. But for foreign colorspaces, the colormatrix is transformed to the image colorspace like described above.

Some final remarks:

The colormatrix multiplication needs all channels for each pixel. Therefore it doesn't work with subsampled chroma planes. An exception is the brightness/contrast/saturation/hue filter, because luminance and chroma operations are completely separated here.
For integer pixelformats the floating point matrix is converted to an integer matrix, where the actual ranges/offsets for Y'CbCr are taken into account.
In practically all cases, the color values can over- or underflow. The processing routines must do proper clipping.

Back from California

2008-09-21T19:44:00.003+02:00

My job sometimes requires me to attend international conferences and workshops. It's pretty exciting, but also very hard work. You have to prepare presentations, posters and articles for the conference digest. You must concentrate on others presentations (ignoring your jet lag) and the brain gets much more input per day than usual. Some people, who wish me a nice holiday before I leave, don't really know what this is about :)

This year, I had the honor to make a 2 weeks trip to Southern California, first to San Diego and then to Pasadena. It was my first visit to the US so here are some differences I noticed with respect to Old Europe. Of course these are highly subjective and based on what I saw in two weeks in just a tiny corner of a huge country.

Shops, Restaurants
If you are used to the unfriendlyness of German shop employees and waiters, you'll be positively surprised in the US. They always have some friendly words for you. They might not be serious with that, but they do it well enough so it works. This is definitely something, where Germans can learn from the US.

Public transport
As a resident of a big German city, I can live perfectly without owning a car. Of course people here always complain about the local trains being too expensive, finishing operation too early in the night etc. But this is still paradise compared the US. At the bus stops in San Diego I saw mostly people who didn't really look prosperous. It seems that everyone who can afford a car, buys one. For a reason.

International news
If you want to learn about a foreign society it's a good idea to watch their TV programs. I learned that American TV news (aside from being extremely hysteric) mostly deal with domestic issues. I talked to an American colleague about that. I was quite surprised that he told me exactly, what I already thought: Americans are self-centered. Maybe a bit of information about foreign countries and societies (especially the ones you plan to bomb) would make some things go more smoothly.

Freedom
Both Western European societies and the US appreciate personal freedom. But the definitions of freedom seem to be somewhat different. In Europe you can usually drink alcohol in public and on many beaches you can decide yourself how much you wear. Americans want to be able to buy firearms, drive big cars and put their dishes into the trashcan after eating. In Europe, I don't miss any of the American freedoms. Not sure about the vice versa.

Shows
Shows of any kind in the US have definitely another dimension. Germans seem to be way too modest to do something like the killer-whale show in the San Diego Seaworld or the shows in the Universal studios in Hollywood.

Image transformation

2008-09-07T19:31:00.015+02:00

Lots of graphics and video effects can be described by a coordinate transformation: There is a rule for calculating destination coordinates from the source coordinates. The rule completely describes the type of transformation.

The most prominent family of transformations are the affine transforms, where the transform rule can be described by a vector matrix multiplication:

(x_dst, y_dst) = (A) (x_src, y_src)

With this one can implement e.g. scaling and rotation. If the vectors and matrix are given in homogeneous coordinates, one can also shift the image by a vector. Other transforms can be Lens distortion, perspective distortion, wave effects and much more.

Now the question: how can such a transform be implemented for digital images? The algorithm is straightforward:

Find the inverse transform. For affine transforms, it will simply be the inverse matrix. For other transforms it might not be that easy. The inverse transform will give you the source coordinates as a function of the destination coordinates
For each destination pixel, find the coordinates in the source image
If the source coordinates are fractional (they usually will be), interpolate the destination pixel from the surrounding source pixels

Implementation

Since the interpolation is always the same I decided to implement this in a generic way in gavl (gavl_image_transform_t). With this we can implement lots of different transforms by just defining the inverse coordinate transform as a C-function and passing it to the interpolation engine. While the algorithm is simple in theory the implementation has to take care for some nasty details:

1. Y'CbCr formats with subsampled chroma planes
In the video area, these are the rule rather than the exception. Many filters don't support them and use an RGB format instead (causing some conversion overhead). They can, however, be handled easily if you set up separate interpolation engines for each plane. For the subsampled planes, you do the following:

Transform the coordinates of the chroma location to image coordinates (i.e. multiply by the subsampling factors and shift according to the chroma placement)
Call the function to get the source coordinates the usual way
Transform the source coordinates back to the coordinates of the chroma plane

2. Destination pixels which are outside the source image
This is e.g. the case where an image is downscaled. Gavl handles these by not touching the destination pixel at all. Then you can fill the destination frame with a color before the transformation and this color will be the background color later on.

3. Destination pixel is inside the source image, but surrounding pixels (needed for interpolation) are not
Here, one can discuss a lot what should be done. For gavl, I decided to assume, that the "missing pixels" have the same color as the closest border pixel. The reason is, that instead of handling all possible cases inside the conversion loop for each pixel (which will slow things down due to the additional branches), one can simply shift the source indices and modify the interpolation coefficients once during initialization. The following figure illustrates, how this is done:

The start index n and the interpolation coefficients are saved in the interpolation table. After shifting the table, the interpolation routine works without branches (and without crashes). Due to the way the interpolation coefficients are modified we assume that the missing pixels at -2 and -1 are the same color as the border pixel at 0. Of course this is done for x and y directions and also for the case that indices are larger than the maximum one.

Usage

The image transformation is very easy to use, just get the gavl from CVS and read the API documentation. There is also a gmerlin filter in CVS (fv_transform) which can be used as reference. Some questions might however arise when using this:

1. How exactly are coordinates defined?
Gavl scaling and transformation routines work with subpixel presision internally. This is necessary, if one wants to handle chroma placement correctly. To make everything correct one should think a bit how coordinates are exactly defined. This is an example for a 3x3 image:

The sample values for each pixel are taken from the pixel center. This means, the top-left pixel has a color value corresponding to the location (0.5, 0.5). For chroma planes, the exact sample locations are considered as described here.

2. Handling of nonsquare pixels
These must be handled by the coordinate transform routine provided by you. Basically, you have a "sample aspect ratio" (sar = pixel_width / pixel_height). In your transformation function, you do something like:


x_dst *= sar; /* Distorted -> undistorted */

/* Calculate source coordinate assuming undistorted image */

x_src /= sar; /* Undistorted -> distorted */

3. Image scaling
One is tempted to think, that this all-in-one solution can be used for scaling as well. It is, of course true, but it's a stupid thing to do. Scaling can be highly optimized in many ways. The gavl_video_scaler_t does this. It's thus many times faster than the generic transform.

4. Downsampling issues
The image transform makes no assumptions about the type of the transform. Especially not if the transform corresponds to downsampling or not. And this is where some issues arise. While for upsampling it's sufficient to just interpolate the destination pixels from the source pixels, for downsampling the image must be low-pass filtered (i.e. blurred) first. This is because otherwise the sampling theorem is violated and aliasing occurs. A very scary example for this is discussed here. One more reason to use the gavl_video_scaler_t wherever possible because it supports antialiasing filters for downscaling. The good news is that usual video material is already a bit blurry and aliasing artifacts are hardly visible.

Examples
After this much theory, finally some examples. These images were made with the gmerlin transform filter (the original photo was taken in the South Indian ruin city of Hampi).

Perspective effect. Coordinate transform ported from the Gimp.

Rotation

Whirl/pinch (Coordinate transform ported from cinelerra)

Lens distortion (Coordinate transform ported from EffecTV)

Generic affine (enter single matrix coefficients)

Music from everywhere - everywhere

2008-09-07T00:03:00.004+02:00

Project goal was simple. Available audio sources are

Vinyl records
Analog tapes
CDs
Radio (analog and internet)
Music files in all gmerlin supported formats on 2 PCs

All these should be audible in stereo in all parts of the apartment including balcony and bathroom. Not everywhere at the same time though and not necessarily in high-end quality.

This had been on my wish list for many years, but I was too lazy to lay cables through the whole apartment (especially through doors, which should be lockable). And since there are lots of signal paths, the result would have been a bit messy. Dedicated wireless audio solutions didn't really convince me, since they are mostly proprietary technology. I never want to become a victim of Vendor lock-in (especially not at home).

When I first read about the WLAN-radios I immediately got the idea, that those are the key to the solution. After researching a lot I found one radio which has all features I wanted:

Stereo (if you buy a second speaker)
Custom URLs can be added through a web interface (many WLAN radios don't allow this!)
Ogg Vorbis support (so I can use Icecast2/ices2 out of the box)

The block diagram of the involved components is here:

Now I had to set up the streaming servers. The icecast server itself installs flawlessly on Ubuntu 8.04. It's started automatically during booting. For encoding and sending the stream to icecast, I use ices2 from the commandline. 2 tiny problem had to be solved:

Ubuntu 8.04 uses PulseAudio as the sound server, while ices2 only supports Alsa. Recording from an Alsa hardware device while PulseAudio is running doesn't work.
For grabbing the audio from a running gmerlin player the soundcard and driver need to support loopback (i.e. record what's played back). This is the case for the Audigy soundcard in the Media PC, but not for the onboard soundcard in the desktop machine.

Both problems can be solved by defining pulse devices in the ~/.asoundrc file:

pcm.pulse
{
type pulse
}

pcm.pulsemon
{
type pulse
device alsa_output.pci_8086_293e_sound_card_0_alsa_playback_0.monitor
}

ctl.pulse
{
type pulse
}

The Alsa devices to be written into the ices2 configuration files are called pulse and pulsemon. The device line is hardware dependent. Use the PulseAudio Manager (section Devices) to find out the corresponding name for your card. If your card and driver support loopback, the pulsemon device isn't necessary.

Some fine-tuning can be done regarding encoding bitrate, buffer sizes and timeouts. When I optimized everything for low latency, icecast considered the WLAN radio too slow and disconnected it. More conservative settings work better. Encoding quality is always 10 (maximum), the corresponding bitrate is around 500 kbit/s.

Mission accomplished

gcc visibility

2008-09-06T01:36:00.004+02:00

Newer versions of gcc (e.g. gcc-4.2.3 coming with Ubuntu-8.04) support a feature called visibility. It is explained in detail here. In essence, it lets you declare functions only within the scope of a shared library. The advantages are:

Since these functions are not callable from outside, the dynamic symbol table becomes smaller. This speeds up loading (especially in the debugger I assume) and makes a smaller library.

If gcc knows at compile-time, that a called function will be in the same shared lib, it can optimize the function call.

My test candidate for this was libquicktime, because it contains lots of private functions (e.g those in lqt_funcprotos.h). Supporting visibility is easy, this is what I did:

Let the configure script append -fvisibility=hidden to the CFLAGS if that's supported by gcc.

Put the function declarations in public headers between

#pragma GCC visibility push(default)

and

#pragma GCC visibility pop.

If there are functions called from outside (e.g. from libquicktime codecs) declared in private headers, add

__attribute__ ((visibility("default")))

to their declarations.

In the libquicktime case, this change also revealed some dirty corners (which I cleaned up after compiling with -Wmissing-declarations).

Let's see, what actually changed:

1. Entries in the dynamic symbol table:
# nm -D /opt/gmerlin/lib/libquicktime.so.0.0.0 | grep " T " | wc -l
Before: 866
After: 278

2. Size of the shared library (compiled with standard settings):
Before: 622K
After: 564K

I did some rough speed tests with time qtinfo some_file.mov, but the differences were too little and the measurement method too inaccurate to be significant.

Let's see how this settles down. Doing the same for the gmerlin libraries will be much easier, because they are much cleaner already :)

Make an analog 5.1 switch using scart components

2008-09-05T01:14:00.006+02:00

The project had 3 clear goals:

1. Remove the disastrous cable mess around the Hifi-system and TV

2. Avoid repeatedly creeping into dark corners to reconnect the cables (always increasing the mess mentioned in 1)

3. Do 1 and 2 without decreasing functionality of the whole system

After considering all necessary signal-paths, it turned out, that one component was missing: An analog 5.1 switch. My 5.1 receiver (from Yamaha) is almost 10 years old now, but it still works perfectly. Unfortunately, it has only one 5.1 input (and lots of stereo inputs). Now, I have 2 sources of 5.1 audio: The DVD player and the Media PC.

And to make things worse: Many stereo movies on the PC (e.g. rips from VHS tapes) are encoded with the old Dolby Surround method. The Yamaha receiver handles them perfectly, but only when the signal comes into a stereo input. This means that the stereo channels form the 5.1 signal must also be routed to a stereo input of the receiver.

So I was searching for something, which does the same as the green box in the following figure (the AUX channel is reserved for future extensions):

Googling around revealed only one solution, which is unacceptable because it only switches between 2 sources, needs a power supply (violating goal 1) and costs € 258. All other search results were people saying that this sucks.

The question was, if some other device can be tricked into doing exactly what I want, and the answer was yes. The idea is to (ab)use the video wires of scart-cables for the additional audio channels. So I got 3 adapters with one scart connector and 6 cinch connectors each:

And a scart switch-box:

The funny thing about the switch box is, that the stereo channels are available at additional cinch connectors (right side), so I can route front L/R to a stereo input of my receiver, like I wanted.

Mission accomplished.

The future from the last century

2008-09-04T21:44:00.003+02:00

When I was younger (in the 70s), I spent a lot of time reading books about the exciting things to come in the year 2000 and beyond. We were supposed to have at least one permanent moon base, public transport with Cabin taxis and underwater cities. And of course peace, happiness and prosperity all over the planet.

Today, it's a bit disappointing to see that none of these things actually became true. Humans seem to be most innovative when it comes to better weapons or more effective Vendor lock-in.

Technical advances must go hand in hand with advances of the human mind; everything else is doomed to failure. Maybe the internet (which was not mentioned in my children's books) can help here.