Gmerlin prereleases

Check these out:

http://gmerlin.sourceforge.net/gmerlin-dependencies-20090606.tar.bz2
http://gmerlin.sourceforge.net/gmerlin-all-in-one-20090606.tar.bz2

If you want to help pushing things forward, try to install and run them and report any problems you encountered.

What evildoers do, if the web gets censored

Lots of discussion happens in German cyberspace around the planned blocking of child porn. It is foreseen to redirect DNS queries for blacklisted domains to a "Stop" sign, which will tell the user, that the site (s)he wanted to visit is blocked. Contrary to earlier promises the IP addresses of the users will be made available to the law enforcement agencies for what they call real time surveillance.

The proponents (one of them confused "servers" and "surfers" in a parliament debate) try their best to make every opponent of this plan look like a pedophile.

I think it's the opposite: Child molesters and other evildoers will have a lot of fun once the blocking mechanism is operational.

Disclaimer
I neither plan nor endorse any of the activities listed below. I only describe theoretical possibilities to demonstrate, how easy it is to abuse such a nonsense.

Things to come

• Pedophile newbies, who search for child porn, will find a leaked block-list from another country. The secret German list will leak as well, sooner or later. It was never as easy as today.
  
• Less smart pedophiles will google for ways to circumvent the blockade. The first hit will be some commercial "surf anonymous" proxy provider. In the end, even more people will earn money with abused kids.
  
• Smarter pedophiles will google better and configure another DNS server. If using another DNS server becomes illegal, they'll find a free SSL proxy. If SSL proxies become illegal, they'll use foreign VPN provider. Connections to foreign VPN providers cannot be illegal because they are used by lots of big companies.
  
• Less smart child porn webmasters will simply use numeric IPs for linking to their sites
  
• Smarter webmasters will regularly send a DNS query for their domain to a censored DNS server. If the returned IP number becomes incorrect, it's time to shut down the site and reopen it somewhere else under a different name. Can easily be scripted into a fully automated warning system. Also works if the block-list is really secret.
  
• Never give your IP address to a bad guy. Today you can get hacked, if you have a poor firewall. Tomorrow the cops might pay you a visit. There is no firewall against cops wanting to visit you. That's because one can easily flood the DNS server of your German provider with forged requests for some blocked domain and your IP address as source. Should be no more than 50 lines in C, including comments.
  
• The ultimate end of civilization is near if the first stop-sign worm appears. Not only because it'll DDOS the stop-sign servers. Most important effect will be to flood the law enforcement agencies with IPs of innocent people. Less time to concentrate on the bad guys.
  

Those, who know the facts and still want to install DNS blockades against child porn, are either pedophiles, criminals or conspirators who'd like a censorship mechanism for all kinds of unpleasant content. The latter might be the reason why each country maintains it's own block list.

Alternatives
Everyone except the pedophiles supports a serious fight against child abuse and child porn, not only in the internet. There are lots of organizations, who are committed to this. Interestingly, some of them are against blockades. Here are 3 German ones:

Carechild
Abuse victims against internet blockades
Self-help group of abused women

Psychedelic visualization programming howto

A 3 day project was to enhance gmerlins default visualization (the one which comes with the gmerlin tarball) a bit. Here is a description, how this visual work. There are lots of other visuals out there, which are based on the same algorithm.

General alrorithm
It's actually pretty simple: Each frame is the last frame filtered a bit and with something new drawn onto it. This repeats over and over.

Drawing something
There are 2 absolutely trivial things to draw, which are directly generated by the audio signal:

• Stereo waveform: Looks pretty much like a 2 beam oscilloscope
  
• Vectorscope: One channel is the x-coordinate, the other one is the y-coordinate. The whole thing is then rotated by 45 degrees counterclockwise. It only sucks for mono signals, because it will display a pretty boring vertical line then.
  

The vectorscope is drawn as lines and as dots. This makes a total of 3 foreground patterns in 7 colors.

Filtering
There are 3 types of filters, which are always applied to the last frame: An image transformation, a colormatrix and a 3-tap blur filter. The blur filter is needed to wipe out artifacts when doing the image transform repeatedly on the same image.

The image transform uses the (usually bad) linear interpolation, which means that the recursive process also blurs the image. There are 8 transforms: Rotate left, rotate right, zoom in, zoom out, ripple, lens, sine and cosine.

Then there are 6 colormatrices. Each of these fades 1 or 2 of the RGB components to zero. Together with the foreground colors (which always have all 3 components nonzero) we get a large variation of the colors when they are recursively filtered.

Background colors
The image transformation sometimes makes areas in the current frame, which are outside of the last frame. These areas are filled with one of 6 background colors. Background colors are dark (all components < 0.5) to increase the contrast against the foreground.

Beat sensitivity
Then I wrote the most simple beat detection: For each audio frame, compute the sound energy (i.e. calculate the sum of the squared samples for each channel). If the energy is twice as large as the time averaged energy, we have a beat. The time averaged energy if calculated by doing a simple RC low-pass filtering of the energies for each frame:

avg_energy = avg_energy * 0.95 + new_energy * 0.05;

Variations
The following things can be changed on each beat (and a random decision with defined probability):

• Foreground pattern (3)
  
• Background color (5)
  
• Transform (8)
  
• Colormatrix (6)
  

This makes 5040 principal variations.

Result
I hacked a small tool (gmerlin_visualize), which takes an audio file and puts it into a video file together with the visualization. The result is here:

Simple music visualization from gmerlin on Vimeo.

As you can see, the video compression on vimeo isn't really suited for that kind of footage. But that's not vimeos fault. It's a general problem that video compression algorithms are optimized for natural images rather than computer generated images.

Possible improvements
The variety can easily be extended by adding new image transformations and new foreground patterns. Also, some contrast between foreground and background is sometimes a bit weak. A more careful selection of the colors and colormatrices would bring less variety but a nicer overall appearance.

Feel free to make a patch :)

Lessons from LAC

I was at the Linux Audio Conference in Parma/Italy. The weather sucked most of the time, but everything else was great. These are the most important things I learned:

Jack rules
Of course I already heard about the Jack sound system before. But since I never had a linux distribution, which installs jack by default, I didn't pay much attention to it.

If you go to the LAC, you start to think that there is hardly any audio architecture besides jack. After looking a bit into Jack (and writing gmerlin plugins for Jack audio I/O) I learned, that it's actually pretty cool. I like especially the way, you can connect arbitrary jack applications together while they are running. This is a nice feature for the gmerlin-visualizer since it relies on grabbing the audio stream from another application.

Ambisonics rocks
I also heard about Ambisonics before but mostly ignored it. If you read the Wikipedia article, you get the impression, that ambisonics is a quite exotic recording/playback technique, which is used only by a small group of enthusiasts.

If you go to LAC, you get the impression that Ambisonics is the standard and things like stereo or 5.1 are quite exotic recording/playback techniques, which are used only by a small group of enthusiasts.

Audio vs. video development
I told the audio developers that they don't know how lucky they are. They agreed on an audio capture/playback architecture (jack), they all use the same internal format (floating point, not interleaved) for processing and they manage to make a conference each year.

In the video area, things are much worse. Capture and playback APIs exist, but they are far too many. Some people do internal processing in RGB, others prefer Y'CbCr. Converting to a common internal format is not feasible because video has much higher data rates than audio and conversion overhead must always be minimized. Also the video developer community is very fragmented, centered around different frameworks (each one being the standard). Cross project communication is sparse. The only common code is ffmpeg for video en-/decoding, but mostly because nobody wants to reimplement such a beast.

Gmerlin development
I talked a lot with Richard Spindler from the openmovieeditor, which makes heavy use of gmerlin libraries. We agreed that our collaboration works quite well because the development activities of gmerlin and openmovieeditor are nicely decoupled. Let's proceed like we already did.

Dirac video

Libquicktime and gmerlin-avdecoder now support Dirac in quicktime. En- and decoding is done with the libschrödinger library. Having already implemented support for lots of other video codecs I noticed some things, both positive and negative.

Positive

• Very precise specification of the uncompressed video format. Interlacing (including field order) is stored in the stream as well as singal ranges (video range or full range). This brings direct support for lots of colormodels.
  
  
• Support for > 8 bit encoding. This is really a rare feature. While ffmpeg always sticks with 8 bit even for codecs with 10 bit or 12 bit modes, the libschrödinger API has higher precision options. Not sure if these modes are really supported internally by now.
  
  
• It seems to aim for scalability from low-end internet downloads to intra-only modes for video editing applications. A lossless mode is also there. Whether it performs equally well for all usage scenarios is yet to be found out.
  
  
• It pretends to be patent free. But since the patent jungle is so dense, it is almost impossible to prove this.
  
  
• It has the BBC behind it, which hopefully means serious development, funding and a chance for a wide deployment.
  

Negative

Sequence end code in an own sample
In the quicktime mapping specification it is required, that the sequence end code (a 13 byte string telling that the stream ends here) must be in an own sample. This is a mess, since for all Quicktime codecs I know (even the most disgusting ones) 1 sample always corresponds to 1 frame. Having a sample, which is no frame, screws up the timing because there is a "duration" associated with the sequence-end-sample, which makes the total stream duration seem larger than is actually is. Also, a frame accurate demuxer will expect one frame more than the file actually has. For both libquicktime and gmerlin-avdecoder I wrote workarounds for this (they simply discard the last packet).

If I had written the mapping spec, I would require the sequence end code to be appended to the last frame (in the same sample). In addition it can be made optional since it's not really needed in quicktime.

If libquicktime encodes dirac files, everything is done according to the spec. Conformance to the spec is more important than my personal opinion about it :)

No ctts atom required
Quicktime timestamps (as given by the stts atom) are decoding timestamps. For all low-delay streams (i.e. streams without B-frames), these are equal to the presentation timestamps. For H.264 and MPEG4 ASP streams with B-frames, the ctts atom specifies the difference between PTS and DTS for each frame and lets the demuxer calculate correct presentation timestamps without touching the video data. If the ctts atom is missing, such quicktime files become as disgusting as AVIs with B-frames. Unfortunately the ctts atom isn't required by the mapping spec, which means we'll see such files in the wild.

The good news is, that the ctts atom isn't mentioned at all in the spec. From this I conclude that it is not forbidden either. Therefore, libquicktime always writes a ctts atom if the stream has B-frames.

On the decoding side (in gmerlin-avdecoder), the quicktime demuxer checks if a dirac stream has a ctts atom. If yes, it is used and everything is fine. If not (i.e. if the file wasn't written by libquicktime), a parser is fired up and an index must be built if one wants sample accuracy. The good news is that the parser is pretty simple and the same thing is needed anyway for decoding dirac in MPEG transport streams.

Other news

gmerlin-avdecoder also supports Dirac in Ogg and MPEG-2 transport streams.

Parallelizing video routines

Basics
One project shortly after the last gmerlin release was to push the CPU usage beyond 50 % on my dual core machine. There is lots of talk about parallelization of commonly used video routines, but only little actual code.

In contrast to other areas (like scientific simulations) video routines are extremely simple to parallelize because most of them fulfill the following 2 conditions:

• The outermost loop runs over scanlines of the destination image
  
• The destination image is (except for a few local variables) the only memory area, where data are written
  

So the first step is to change the conversion functions of the type:

void conversion_func(void * data, int width, int height)
{
int i, j;
for(i = 0; i < height; i++)
  {
  for(j = 0; j < width; j++)
    {
    /* Do something here */
    }
  }
}

to something like:

void conversion_func(void * data, int width, int start, int end)
{
int i, j;
for(i = start; i < end; i++)
  {
  for(j = 0; j < width; j++)
    {
    /* Do something here */
    }
  }
}

The data argument points to a struct containing everything needed for the conversion like destination frame, source frame(s), lookup tables etc. The height argument was replaced by start and end arguments, which means that the function now processes a slice (i.e. a number of consecutive scanlines) instead of the whole image. The remaining thing now is to call the conversion function from multiple threads. It is important that the outermost loop is split to keep the overhead as small as possible.

The API perspective
Everything was implemented

• With a minimal public API
  
• Backwards compatible (i.e. if you ignore the API extensions things still work)
  
• Completely independent of the actual thread implementation (i.e. no -lpthread is needed for gavl)
  

The whole magic can be boiled down to:

/* Slice process function, which is passed by gavl to the application */
typedef void (*gavl_video_process_func)(void * data, int start, int end);

/* Routine, which passes the actual work to the worker thread (which is managed by the application) */
typedef void (*gavl_video_run_func)(gavl_video_process_func func,
                               void * gavl_data,
                               int start, int end,
                               void * client_data, int thread);

/* Synchronization routine, which waits for the worker thread to finish */
typedef void (*gavl_video_stop_func)(void * client_data, int thread);

/* Set the maximum number of available worker threads (gavl might not need all of them
if you have more CPUs than scanlines) */
void gavl_video_options_set_num_threads(gavl_video_options_t * opt, int n);

/* Pass the run function to gavl */
gavl_video_options_set_run_func(gavl_video_options_t * opt,
                                gavl_video_run_func func,
                                void * client_data);

/* Pass the stop function to gavl */
void gavl_video_options_set_stop_func(gavl_video_options_t * opt,
                                 gavl_video_stop_func func,
                                 void * client_data);

Like always, the gavl_video_options_set_* routines have corresponding
gavl_video_options_get_* routines. These can be used for using the same
multithreading mechanism outside gavl (e.g. in a video filter).

The application perspective
As noted above there is no pthread specific code inside gavl. Everything is passed via callbacks. libgmerlin has a pthread based thread pool, which does exactly what gavl needs.

The thread pool is just an array of context structures for each thread:

typedef struct
{
/* Thread handle */
pthread_t t;

/* gavl -> thread: Do something */
sem_t run_sem;

/* thread -> gavl: I'm done */
sem_t done_sem;

/* Mechanism to make the fuction finish */
pthread_mutex_t stop_mutex;
int do_stop;

/* Passed from gavl */
void (*func)(void*, int, int);
void * data;
int start;
int end;
} thread_t;

The main loop (passed to pthread_create) for each worker thread looks like:

static void * thread_func(void * data)
{
thread_t * t = data;
int do_stop;
while(1)
  {
  sem_wait(&t->run_sem);

  pthread_mutex_lock(&t->stop_mutex);
  do_stop = t->do_stop;
  pthread_mutex_unlock(&t->stop_mutex);

  if(do_stop)
    break;
  t->func(t->data, t->start, t->len);
  sem_post(&t->done_sem);
  }
return NULL;
}

The worker threads are launched at program start and run all the time. As long as nothing is to do, they wait for the run_sem semaphore (using zero CPU). Launching new threads for every little piece of work would have a much higher overhead.

Passing work to a worker thread happens with the following gavl_video_run_func:

void bg_thread_pool_run(void (*func)(void*,int start, int len),
                     void * gavl_data,
                     int start, int end,
                     void * client_data, int thread)
{
bg_thread_pool_t * p     = client_data;
p->threads[thread].func  = func;
p->threads[thread].data  = gavl_data;
p->threads[thread].start = start;
p->threads[thread].end   = end;

sem_post(&p->threads[thread].run_sem);
}

Synchronizing (waiting for the work to finish) happens with the following gavl_video_stop_func:

void bg_thread_pool_stop(void * client_data, int thread)
{
bg_thread_pool_t * p     = client_data;
sem_wait(&p->threads[thread].done_sem);
}

The whole thread pool implementation can be found in the gmerlin source tree in lib/threadpool.c (128 lines including copyright header).

Implementations
Parallelization was implemented for

• The video scaler
  
• The colormatrix filters
  
• The Image transformation
  
• The Yadif port in the deinterlacing filter
  

Benchmarks
For benchmarking one needs a scenario where the parallelized processing routine(s) need the lions share of the total CPU usage. I decided to make a (completely nonsensical) gaussian blur with a radius of 50 over 1000 frames of a 720x576 PAL sequence. All code was compiled with default (i.e. optimized) options. Enabling 2 threads descreased the processing time from 81.31 sec to 43.35 sec.

Credits
Finally found this page for making source snippets in blogger suck less.

Elementary stream parsing

See the image below for an updated block-schematics of gmerlin-avdecoder as addition to my last blog entry:



You see, what's new here: A parser, which converts "bad" packets into "good" packets. Now what does that exactly mean? A good (or well formed) packet has the following properties:

• It always has a correct timestamp (the presentation time at least)
  
• It has a flag to determine if the packet is a keyframe
  
• It has a valid duration. This is necessary for frame accurate repositioning of the stream after seeking. At least if you want to support variable framerates.
  

Good packets (e.g. from quicktime files) can directly be passed to the decoder, and it's possible to build an index from them. Unfortunately some formats (most notably MPEG program- and transport streams), don't have such nice packets. You neither know where a frame starts, nor whether it is a keyframe. Large frames can be split across many packets, some packets can contain several small frames. Also timestamps are not available for each frame. To make things worse, these formats are very widely used. You find them in .mpg files, on DVDs, (S)VCDs, BluRay disks and in DVB broadcasts. Also all newer formats for consumer cameras (HDV and AVCHD) use MPEG-2 transport streams. Since they are important for video editing applications, so sample accurate access is a main goal here.

The first solutions for this were realized inside the decoders. libmpeg2 is very tolerant with regard to the frame alignment and ffmpeg has an AVParser, which splits a continuous stream into frames. Additional routines were written for building an index.

It was predictable that this would not be the ultimate solution. The decoders got very complicated and building the index was not possible without firing up an ffmpeg decoder (because the AVParser doesn't tell about keyframes) so index building was very slow.

So I spent some time to write a parsers for elementary A/V streams, which parse the streams to get all necessary information for creating well formed packets.

After that worked, I could be sure, that every codec always gets well-formed packets. What followed then, was by far the biggest cleanup in the history of gmerlin-avdecoder. Many things could be simplified, lots of cruft got kicked out, duplicate code got moved to central locations.

New features are:

• Decoding of (and sample accurate seeking within) elementary H.264 streams
  
• Sample accurate access for elementary MPEG-1/2 streams
  
• Sample accurate access for MPEG program streams with DTS- and LPCM audio
  
• Faster seeking because non-reference frames are skipped while approaching the new stream position
  
• Much cleaner and simpler architecture.
  

The cleaner architecture doesn't necessarily mean less bugs, but they are easier to fix now :)