Fun with the colormatrix

Some years ago, when I wrote lemuria, I was always complaining that OpenGL doesn't have a colormatrix. Later I found, that some extensions provide this, but not the core API. Today I think the main problem was that I used a graphics system (OpenGL) which is optimized to look as realistic as possible for a visualization which should look a surrealistic as possible :)

Later when I concentrated on more serious work like developing high quality video filters, I rediscovered the colormatrix formalism. Now what is that all about and what's so exciting about that? If you assume each pixel to be a vector of color channels, multiplying the vector by a matrix is simply a linear transformation of each pixel. In homogenous coordinates and in RGBA or YUVA (i.e. 4 channel) colorspace, it can be completely described by a 4x5 matrix. Now there are lots of commonly used video filters, which can be described by a colormatrix multiplication:

Brightness/Contrast (Y'CbCrA)

[	c	0	0	0	b - (1+c)/2	]
[	0	1	0	0	0	]
[	0	0	1	0	0	]
[	0	0	0	1	0	]

b and c are between 0.0 and 2.0.

Saturation/Hue rotation (Y'CbCrA)

[	1	0	0	0	0	]
[	0	s*cos(h)	-s*sin(h)	0	0	]
[	0	s*sin(h)	s*cos(h)	0	0	]
[	0	0	0	1	0	]

h is between -pi and pi (0 is neutral). s is between 0.0 (grayscale) and 2.0 (oversaturated).

Invert single RGB channels (RGBA)

[	-1	0	0	0	1	]
[	0	1	0	0	0	]
[	0	0	1	0	0	]
[	0	0	0	1	0	]

This inverts only the first (red) channel. Inverting other channels is done by changing any of the other lines accordingly.

Swap RGB channels (RGBA)

[	0	0	1	0	0	]
[	0	1	0	0	0	]
[	1	0	0	0	0	]
[	0	0	0	1	0	]

This swaps the first (red) and 3rd (blue) channel. It can be used to rescue things if some buggy routine confused RGB and BGR. Other swapping schemes are trivial to do.

RGB gain (RGBA)

[	gr	0	0	0	0	]
[	0	gg	0	0	0	]
[	0	0	gb	0	0	]
[	0	0	0	1	0	]

g_r, g_g and g_b are in the range 0.0..2.0.

There are of course countless other filters possible, like generating an alpha channel from (inverted) luminance values etc. Now, what do you do if you want to make e.g. a brightness/contrast filter working on RGBA images? The naive method is to transform the RGBA values to Y'CbCrA, do the filtering and transform back to RGBA. And this is what can be optimized in a very elegant way by using some simple matrix arithmetics. Instead of performing 2 colorspace conversions in addition to the actual filter for each pixel, you can simply transform the colormatrix (M) from Y'CbCrA to RGBA:

Untransformed pixel in RGBA:	(p)
Untransformed pixel in Y'CbCrA:	(RGBA->Y'CbCrA)*(p)
Transformed pixel in Y'CbCrA:	(M)*(RGBA->Y'CbCrA)*(p)
Transformed pixel in RGBA:	(Y'CbCrA->RGBA)*(M)*(RGBA->Y'CbCrA)*(p)

The matrices (RGBA->Y'CbCrA) and (Y'CbCrA->RGBA) are the ones which convert between RGBA and Y'CbCrA. Since matrix multiplication is associative you can combine the 3 matrices to a single one during initialization:

(M)_RGBA = (Y'CbCrA->RGBA)*(M)_Y'CbCrA*(RGBA->Y'CbCrA)

If you have generic number-crunching routines, which to the vector-matrix multiplication for all supported pixelformats, you can reduce conversion overhead in your processing pipeline significantly.

Another trick is to combine multiple transformations in one matrix. The gmerlin video equalizer filter does this for brightness/contrast/saturation/hue rotation.

A number of gmerlin filters use the colormatrix method. Of course, for native pixelformats (image colorspace = matrix colorspace), the transformation is done directly, since it usually needs much less operations than a generic matrix-vector multiplication. But for foreign colorspaces, the colormatrix is transformed to the image colorspace like described above.

Some final remarks:

• The colormatrix multiplication needs all channels for each pixel. Therefore it doesn't work with subsampled chroma planes. An exception is the brightness/contrast/saturation/hue filter, because luminance and chroma operations are completely separated here.
  
• For integer pixelformats the floating point matrix is converted to an integer matrix, where the actual ranges/offsets for Y'CbCr are taken into account.
  
• In practically all cases, the color values can over- or underflow. The processing routines must do proper clipping.