Truths or lies - decide yourself

Monday, December 14, 2009

Flash-free live web video solution

Some time ago, we all knew that video on the web was equivalent to the proprietary Flash technology. Also I used to say, that there might be political or psychological reasons for using Ogg/Theora, but never technical ones.

Well, the conditions have changed recently so it's time to bring an update on this.

HTML 5 video tag
The HTML 5 draft supports a <video> tag for embedding a video into a webpage as a native html element (e.g. without a plugin). Earlier versions of the draft even recommended that browsers should support Ogg/Theora as a format for the video. The Ogg/Theora recommendation was then removed and a lot of discussion was started around this. This wikipedia article summarizes the issue. Nevertheless, there are a number of browsers supporting OggTheora video out of the box, among these is Firefox-3.5.

Cortado plugin
In a different development, the Cortado java applet for theora playback was written. It is GPL licensed and you can just download it and put it into your webspace.

Now the cool thing about the <video> tag is, that browsers, which don't know about it, will display the contents between <video> and </video>, so you can include the applet code there. Researching a bit about the best way to do this, I read that the (often recommended) <applet> mechanism is not valid html. A better solution is here.

Webmasters side
Now if you have the live-stream at http://192.168.2.2:8000/stream.ogg your html page will look like:

<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
</head>
<body>
<h1>Test video stream</h1>
<video tabindex="0"
src="http://192.168.2.2:8000/stream.ogg"
controls="controls"
alt="Test video stream"
title="Test video stream"
width="320"
height="240"
autoplay="true"
loop="nolooping">
<object type="application/x-java-applet"
   width="320" height="240">
<param name="archive" value="cortado.jar">
<param name="code" value="com.fluendo.player.Cortado.class">
<param name="url" value="http://192.168.2.2:8000/stream.ogg">
<param name="autoplay" value="true">
<a href="http://192.168.2.2:8000/stream.ogg">Test video stream</a>
</object>
</video>
</body>
</html>

Note that for live-streams the "autoplay" option should be given. If not, firefox will try to load the first image (automatically anyway) to show it in the video widget. Then it will stop downloading the live-stream until you click start. Pretty obvious that this will mess up live-streaming.

Server side
For live streaming I just installed the icecast server, which came with my ubuntu. I just changed the passwords in /etc/icecast/icecast.xml, enabled the server in /etc/default/icecast2 and started it with /etc/init.d/icecast2 start.

Upstream side
There are lots of programs for streaming to icecast servers, most of them use libshout. I decided to create a new family of plugins for gmerlin: Broadcasting plugins. They have the identical API as encoder plugins (used by the transcoder or the recorder). The only difference is, that they don't produce regular files and must be realtime capable.

Using libshout is extremely simple:

Create a shout instance with shout_new()
Set parameters with the shout_set_*() functions
Call shout_open() to actually open the connection to the icecast server
Write a valid Ogg/Theora stream with shout_send(). Actually I took my already existing Ogg/Theora encoder plugin and replaced all fwrite() calls by shout_send().

See below for the libshout configuration of the gmerlin Ogg broadcaster.

Of course, some minor details were left out in my overview, read the libshout documentation for them. As upstream client, I use my new recorder application.

The result
See below a screenshot from firefox while it plays back a live stream:

Open Issues
The live-video experiment went extremely smooth. I discovered however some minor issues, which could be optimized away:

Firefox doesn't recognize live-streams (i.e. streams with infinite duration) properly. It displays a seek-slider which always sticks at the very end. Detecting a http stream as live can easily be done by checking the Content-Length field of the http response header.
The theora encoder (1.1.1 in my case) might be faster than the 1.0.0 series, but it's still way too slow. Live encoding of a 320x240 stream is possible on my machine but 640x480 isn't.
The cortado plugin has a loudspeaker icon, but no volume control (or it just doesn't work with my Java installation)

Other news

With the video switched off I can send audio streams to my wlan radio. This makes my older solution (based on ices2) obsolete.
The whole thing of course works with prerecorded files as well. In this case, you can just put the files into your webspace and your normal webserver will deliver them. No icecast needed.

Sunday, December 6, 2009

Introducing Gmerlin recorder

Gmerlin-recorder is a small application, which records audio and video from hardware devices. It was written as a more generic application, which should eventually replace camelot. See below for a screenshot:

As sources, we support:

Audio devices via OSS, Alsa, Esound, Pulseaudio and Jack
Webcams via V4L and V4L2
X11 grabbing

For monitoring you see the video image in the recorder window. For the audio you have a vumeter. With recorded streams you can do lots of stuff:

Filter the streams using gmerlin A/V filters
Write the streams to files using gmerlin encoder plugins
Send the encoded stream to an icecast server (will be described in detail in another blog post)
Save snapshots to images either manually or automatically

With all these features combined in the right way, you can use gmerlin-recorder for a large number of applications:

Make vlogs with your webcam and a microphone
Digitize analog music
Make a video tutorial for your application
Start your broadcasting station
Send music from your stereo to your WLAN radio
Make stop-motion movies
...

Until the recorder can fully replace camelot, we need the following features, which are not there yet:

Fullscreen video display
Forwarding of the video stream via vloopback

Sunday, November 15, 2009

X11 grabbing howto

One gmerlin plugin I always wanted to program was an X11 grabber. It continuously grabs either the entire root window, or a user defined rectangular area of it. It is realized as a gmerlin video recorder plugin, which means that it behaves pretty much like a webcam.

Some random notes follow.

Transparent grab window
The rectangular area is defined by a grab window, which consists only of a frame (drawn by the window manager). The window itself must be completely transparent such that mouse clicks are sent to the window below. This is realized using the X11 Shape extension:

XShapeCombineRectangles(dpy, win,
                     ShapeBounding,
                     0, 0,
                     (XRectangle *)0,
                     0,
                     ShapeSet,
                     YXBanded);

The grab window can be moved and resized to define the grabbing area. This is more convenient than entering coordinates manually. After a resize, the plugin must be reopened because the image size of a gmerlin video stream cannot change within the processing loop.

Make the window sticky and always on top
Depending on the configuration the grab window can always be on top of the other windows. There is also a sticky option making the window appear on all desktops. This is done with the collowing code, which must be called each time before the window is mapped:

if(flags & (WIN_ONTOP|WIN_STICKY))
{
Atom wm_states[2];
int num_props = 0;
Atom wm_state = XInternAtom(dpy, "_NET_WM_STATE", False);
if(flags & WIN_ONTOP)
 {
 wm_states[num_props++] =
   XInternAtom(dpy, "_NET_WM_STATE_ABOVE", False);
 }
if(flags & WIN_STICKY)
 {
 wm_states[num_props++] =
   XInternAtom(dpy, "_NET_WM_STATE_STICKY", False);
 }

XChangeProperty(dpy, win, wm_state, XA_ATOM, 32,
               PropModeReplace,
               (unsigned char *)wm_states, num_props);
}

Grabbing methods
The grabbing itself is done on the root window only. This will grab all other windows inside the grab area. The easiest method is XGetImage, but that allocates a new image with each call. malloc()/free() cycles within the processing loop should be avoided whenever possible. XGetSubImage() allows to pass an allocated image. Much better of course is XShmGetImage(). It was roughly 3 times faster than XGetSubImage() in my tests.

Coordinate correction
If parts of the grabbing rectangle are outside the root window, you'll get a BadMatch error (usually exiting the program), no matter which function you use for grabbing. You must handle this case and correct the coordinates to stay within the root window.

Mouse cursor
Grabbing works for everything displayed on the screen (including XVideo overlays) except the mouse cursor. It must be obtained and drawn "manually" onto the grabbed image. Coordinates are read with XQueryPointer(). The cursor image can be obtained if the XFixes extension is available. First we request cursor change events for the whole screen with

XFixesSelectCursorInput(dpy, root,
                    XFixesDisplayCursorNotifyMask);

If the cursor changed and is within the grabbing rectangle we get the image with

im = XFixesGetCursorImage(dpy);

The resulting cursor image is then converted to a gavl_overlay_t and blended onto the grabbed image with a gavl_overlay_blend_context_t.

Not done (yet)
Other grabbing programs deliver images only when something has changed (resulting in a variable framerate stream). This can be achieved with the XDamage extension. Since the XDamage extension is (like many other X11 extensions) poorly documented, I didn't bother to implement this yet.

One alternative is to use gmerlins decimate video filter, which compares the images in memory. The result will be the same, but CPU usage will be slightly increased.

Saturday, October 31, 2009

gavl color channels

When programming something completely unrelated, I stumbled across a missing feature in gavl: Extract single color channels from a video frame into a grayscale frame. The inverse operation is to insert a color channel from a grayscale frame into a video frame (overwriting the contents of that channel). This allows you to assemble an image from separate color planes.

Both were implemented with a minimalistic API (1 enum and 3 functions), which works for all 35 pixelformats. First of all we have an enum for the channel definitions:

typedef enum
{
GAVL_CCH_RED,    // Red
GAVL_CCH_GREEN,  // Green
GAVL_CCH_BLUE,   // Blue
GAVL_CCH_Y,      // Luminance (also grayscale)
GAVL_CCH_CB,     // Chrominance blue (aka U)
GAVL_CCH_CR,     // Chrominance red (aka V)
GAVL_CCH_ALPHA,  // Transparency (or, to be more precise, opacity)
} gavl_color_channel_t;

For getting the exact grayscale format for one color channel you first call:

int gavl_get_color_channel_format(const gavl_video_format_t * frame_format,
                               gavl_video_format_t * channel_format,
                               gavl_color_channel_t ch);

It returns 1 on success or 0 if the format doesn't have the requested channel. After you have the channel format, extracting and inserting is done with:

int gavl_video_frame_extract_channel(const gavl_video_format_t * format,
                                   gavl_color_channel_t ch,
                                   const gavl_video_frame_t * src,
                                   gavl_video_frame_t * dst);

int gavl_video_frame_insert_channel(const gavl_video_format_t * format,
                                 gavl_color_channel_t ch,
                                 const gavl_video_frame_t * src,
                                 gavl_video_frame_t * dst);

In the gmerlin tree, there are test programs exctractchannel and insertchannel which test the functions for all possible combinations of pixelformats and channels. They are in the gmerlin tree and not in gavl because we load and save the test images with gmerlin plugins.

Saturday, October 24, 2009

Major gmerlin player upgrade

While working on several optimization tasks of the player engine, I found out that the player architecture sucked. So I made a major upgrade (well, a downgrade actually since lots of code was kicked out). Let me elaborate what exactly was changed.

Below you see a block schematics of the player engine as it was before (subtitle handling is omitted for simplicity):

The audio- and video frames were read by the input thread from the file, pulled through the filter chains (see here) and pushed into the fifos.

The output threads for audio and video pulled the frames from the fifos and sent them to the soundcard or the display window.

The idea behind the separate input thread was that if CPU load is high and decoding a frame takes longer than it should, the output threads can still continue with the frames buffered in the fifos. It turned out that this was the only advantage of this approach, and it only worked if the average decoding time was still less than realtime.

The major disadvantage is, that if you have fifos with frames pushed at the input and pulled at the output, the system becomes very prone to deadlocks. If fact, the code for the fifos became bloated and messy over the time.

While programming a nice new feature (updating the video display while the seek slider is moved), the playback was messed up after seeking and I quickly blamed the fifos for this. The resulted was a big cleanup, the result is shown below:

You see, that the input thread and fifos are completely removed. Instead, the input plugin is protected by a simple mutex and the output threads do the decoding and processing themselves. The advantages are obvious:

Much less memory usage (one video frame instead of up to 8 frames in the fifo)
Deadlock conditions are much less likely (if not impossible)
Much simpler design, bugs are easier to fix

The only disadvantage is that if a file cannot be decoded in realtime, audio and video run out of sync. In the old design the input thread took care that the decoding of the streams didn't run too much out of sync. For these cases, I need to implement frame skipping. This can be done in several steps:

If the stream has B-frames, skip them
If the stream has P-frames, skip them (display only I-frames)
Display only every nth I-frame with increasing n

Frame skipping is the next major thing to do. But with the new architecture it will be much simpler to implement than with the old one.

Friday, September 25, 2009

Frame tables

When thinking a bit about video editing, I thought it might be nice to have the timestamps of all video frames in advance, i.e. without decoding the whole file. With that you can e.g. seek to the Nth frame or for a given absolute time you can find the closest video frame even for variable framerate files.

The seeking functions always take a (scaled) time as argument and I saw no reason to change that.

So what I needed was a table which allows the translation of framecounts to/from PTS values even for variable framerate files. For many fileformats a similar information is already stored internally (as a file index) so the task was only to convert the info into something usable and export it through the public API.

Since this feature is very generic and might get used in both gmerlin and gmerlin-avdecoder, I decided to put the stuff into gavl.

Implicit table compression
One very smart feature in the Quicktime format is the stts atom. That's because it stores a table of frame_count/frame_duration pairs. The nice thing is, that for constant framerate files (the vast majority) the table consists of just one entry and translating framecounts to/from timestamps becomes trivial. Only for variable framerate streams, the table has more entries and the translation functions need longer.

The implementation
The frame table is called gavl_frame_table_t and is defined in gavl.h. Note: The structure is public at present but might become private before the next public release.
Here, there are also the translation functions gavl_frame_table_frame_to_time(), gavl_frame_table_time_to_frame() and gavl_frame_table_num_frames().

If you use gmerlin-avdecoder for decoding files, you can use bgav_get_frame_table() to obtain a frame table of a video stream. It can be called after the stream has been fully initialized. Naturally you will want to use sample accurate decoding mode before obtaining the frame table. If you want to reuse the frame table after the file was closed, use gavl_frame_table_save() and gavl_frame_table_load().

Future extension
An also interesting information are the timecodes. First of all, what's the difference between timecodes and timestamps in gmerlin terminology? Timestamps are used to synchronize multiple streams (audio, video, subtitles) of a file with each other for playback. They usually start at zero or another small value and play an important role e.g. after seeking in the files. Timecodes are usually given for each video frame and are used to identify scenes in a bunch of footage. They can resemble e.g. recording time/date.

Timecodes will also be supported by the frame table, but this isn't implemented yet.

Tuesday, September 15, 2009

Back from Malta

After a number of adventure- and/or conference trips, I decided to make a more ordinary vacation. Since the goal was to extend the summer a bit, we decided to travel in the first 2 September weeks (when German weather can already start to suck) and move to the southernmost country, which can be reached without many difficulties. Easy traveling for Europeans means to stay within the EU (and Euro zone), so the destination was chosen to be Malta.

I knew a bit about Malta from TV documentaries, and while reading the Wikipedia article a bit, I became even more curious.

Here are some facts I figured out. For pictures click on the Wikipedia links, they have higher quality than the ones from my crappy mobile.

It was damn hot and humid. Not all people from the Northern countries can withstand that. I was happy to find out that after surviving 4 weeks in the monsoon season in Southern India, I'm more heat resistant than my Greek companion.
It is indeed very stress-free because everyone speaks English (unlike in other Southern European countries) and the islands are small enough to reach practically every destination by bus. Just make sure you stay in the capital Valletta, almost all bus routes start there. On the smaller island Gozo, all busses start in Victoria.
It has an extremely rich 7000 years old history. You can visit prehistoric temples (probably the oldest of their kind worldwide), churches, fortresses and remains from the Romans and Phoenicians. I already saw many churches from the inside before, I was not really a fan of Latin language and Roman history in school, and my companion didn't want to bother looking for sight seeing destinations at all. So I decided to concentrate on the prehistoric stuff.
We saw the temples of Ħaġar Qim, Mnajdra (both covered with giant protective tents now), Tarxien and Ġgantija. The latter ones were the most impressive, especially after we reached them after a long walk in the afternoon heat 20 minutes before the last admission :)
The temples of Skorba are the oldest ones, but not very spectacular and not accessible for the public.
No chance to get into the Hypogeum of Ħal-Saflieni. It was booked out 4 weeks in advance. That sucked.
If you like beaches, Malta is definitely not the #1 destination for you. There are just a few, a nice one we visited is Ramla Bay.
If you like diving at cliffs, Malta is a paradise for you.
Funny is the bit of British culture (Malta was British until 1964). People are driving on the left, you get chips with most meals, and the phone booths and mailboxes are red, like in the UK.
Some Maltese people know more German soccer teams than me.

Shortly after coming back home, German weather started to suck.

Truths or lies - decide yourself

Monday, December 14, 2009

Flash-free live web video solution

Sunday, December 6, 2009

Introducing Gmerlin recorder

Sunday, November 15, 2009

X11 grabbing howto

Saturday, October 31, 2009

gavl color channels

Saturday, October 24, 2009

Major gmerlin player upgrade

Friday, September 25, 2009

Frame tables

Tuesday, September 15, 2009

Back from Malta

Tags

Blog Archive

My projects

About Me

Hits since 25. Jun 2009

Monday, December 14, 2009

Sunday, December 6, 2009

Sunday, November 15, 2009

Saturday, October 31, 2009

Saturday, October 24, 2009

Friday, September 25, 2009

Tuesday, September 15, 2009

Tags

Blog Archive

My projects

Subscribe To

About Me

Hits since 25. Jun 2009