On demand audio streaming with icecast

The project goal was to make the impossible happen: Turn an icecast streaming server into an audio-on-demand server.

The background is, that I bought a NAS, which is basically a PC with Atom CPU. After erasing the firmware and installing Ubuntu Server on a 10 TB Raid 5 system, I was thinking what else I could do with the box.

Live streaming via icecast to my Wifi-radio worked for some time now, but this needs a running PC with a soundcard. What I had in my mind, was different:

• It should run exclusively on the NAS, no need to switch on a PC
  
• It should support an arbitrary number of playlists, each one corresponding to an icecast URL.
  
• The upstream mechanism should work on demand because encoding many mp3 streams in parallel overloads the Atom CPU.
  
• The current song should be shown in the display of the radio
  

Song titles
For the last requirement I added live metadata updating to the API for gmerlin broadcasting plugins. After learning, that Vorbis streams with changing song titles make my radio reboot, I wrote an MP3 broadcasting plugin (with libshout and lame). It seems that later firmware versions for the radio fix the vorbis problem, but the firmware update requires a windows software.

Commandline recorder
The recording and broadcast architecture for gmerlin was already working reliably, so I wrote a plugin, which takes a gmerlin album (=playlist), shuffles the tracks and makes them available as if it record from a soundcard. In addition, I wrote a commandline recorder, which could be started from a script. There is one script for starting a broadcast:

$cat start_broadcast.sh

#!/bin/sh

BITRATE=320
NAME="NAS $1"
STATION_DIR="/nas/Stations/lists/"
PASSWORD="secret"

AUDIO_OPT='do_audio=1:plugin=i_audiofile{album='$STATION_DIR$1':shuffle=1}'
VIDEO_OPT="do_video=0"
METADATA_OPT="metadata_mode=input"
ENC_OPT='audio_encoder=b_lame{server=nas_ip:mount=/'$1':password='$PASSWORD':name='$NAME':cbr_bitrate='$BITRATE'}'

gmerlin-record -aud $AUDIO_OPT -vid $VIDEO_OPT -m $METADATA_OPT -enc "$ENC_OPT" -r 2>> /dev/null >> /dev/null &
echo $! > $1.pid

If you call the script with start_broadcast.sh foo, it will load the album /nas/Stations/lists/foo and send the stream to the icecast server, which will make it available under nas.ip:8000/foo. In addidion, the PID of the process will be written to ./foo.pid so it can be stopped later.

The foo broadcast can be stopped with stop_broadcast.sh foo, where the
script looks like:

#!/bin/sh
kill -9 `cat $1.pid`
rm -f $1.pid

Icecast configuration
No critical options had to be changed in the icecast configuration, except queue-size, which was doubled to 1048576 because it's better for 320 kbps streams.

Icecast stats in awk friendly format
For the on-demand meachism described below, we also need to get the
running channels and connected clients from the server ideally in an awk friendly
format. This is done by getting the server statistics in xml format and process it
with xsltproc, a small commandline tool which comes with libxml2:

$cat get_stats.sh

#!/bin/sh
wget --user=admin --password=secret -O - http://127.0.0.1:8000/admin/stats.xml 2> /dev/null | \
xsltproc stats.xsl - | cut -b 2-

If you have two channels foo (1 listener) and bar (2 listeners) it will output

foo 1
bar 2

The transformation file stats.xsl looks like:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="text"/>
<xsl:template match="/">
 <xsl:for-each select="icestats/source">
    <xsl:value-of select="@mount"/>
    <xsl:text> </xsl:text>
    <xsl:value-of select="listeners"/>
    <xsl:text>
</xsl:text>
 </xsl:for-each>
</xsl:template>
</xsl:stylesheet>

On demand mechanism
Now since we have commands for starting, stopping and querying channels, we can start a channel when the first listener connects and stop it after the last listener disconnected. Since icecast doesn't support on demand streaming, we must trick it into doing so. The idea is to put a second http server in front of the icecast server, which handles the connection requests, starts the channel (if necessary) and then does a http redirect to the real icecast url. The icecast server runs on port 8000, the redirection server (to which the listeners connect) runs on port 8001. The redirection server can be built simply within shell scripts using the netcat (traditional) utility. The server script is simple:

$cat server.sh
#!/bin/sh

cd /nas/mmedia/Stations

while true; do
nc.traditional -l -p 8001 -c ./handle.sh
done

Whenever a TCP connection on port 8001 arrives, the following handler script is executed:

$cat handle.sh
#!/bin/bash

# Read request, path and protocol
read REQ URLPATH PROTO
# Read header variables
while true; do
read VAR VAL
if test "x$VAL" = "x"; then
break
fi
done

# Reject anything but GET requests
if test "x$REQ" != "xGET"; then
echo -e "HTTP/1.1 400 Bad Request\r\n\r\n"
exit
fi

# Remove leading "/"
FILE=`echo $URLPATH | cut -b 2-`

# Close unused streams
./clean.sh $FILE

# Check if we are broadcasting already
RESULT=`./query_station.sh $FILE`
if test "x$RESULT" = "x"; then
./start_broadcast.sh $FILE 2>> /dev/null &
sleep 1
fi

# Send redirection header
URL="http://nas_ip:8000/$FILE"
echo -e "HTTP/1.1 307 Temporary Redirect\r\nLocation: $URL\r\n\r\n"

Here we use 2 additional scripts. clean.sh stops all streams with zero listeners except the one, which was given as commandline argument.

#!/bin/sh
./get_stats.sh | awk -v NAME=$1 '($1 != NAME) && ($2 == 0) { system("./stop_broadcast.sh " $1) }'

query_station.sh lists just the number of listeners of the given station:

#!/bin/sh
./get_stats.sh | awk -v NAME=$1 '$1 == NAME { print $2 }'

Energy saving mode
When we just use the radio, the NAS must be switched on manually. The PCs do that automatically with wake-on-lan. The NAS detects, when it is no longer needed and switches off automatically then. This is done by querying the TCP connections to IP addresses other than localhost. If we don't have any external connections for more than 30 minutes, we switch off. The following script can be interesting for many other applications as well. Simply start it during booting:

#!/bin/sh
# Switch off after this time
THRESHOLD=1800
# Delay between 2 checks
DELAY=60

DATE_START=`date +%s`

while :
do
  CONNECTIONS=`netstat -tn | grep tcp | grep -v " 127\." | wc -l`
  DATE_NOW=`date +%s`

  if test "x$CONNECTIONS" = "x0"; then
    DATE_DIFF=`echo "$DATE_NOW - $DATE_START - $THRESHOLD" | bc`
    if test $DATE_DIFF -gt "0"; then
      poweroff
      exit
    fi
  else
    DATE_START=$DATE_NOW
  fi
  sleep $DELAY
done

Mission accomplished.