BBC Online operations video monitoring wall

I’m Marina Kalkanis and I head up the Core Services teams in BBC Future Media that are responsible for the BBC live and on demand programming on the internet. My colleague Cait O’Riordan has already outlined how the Olympic content – metadata and AV streams - gets to your device, and David Rogers has blogged in detail about building the Olympic data services.

This blog will give you more detail about how we built the media delivery workflow for the online Olympics.

My team takes the live broadcast feeds and turns them into formats that can be delivered on the internet along with the appropriate metadata. We also deliver the Games catchup media that remain available for the next six months.

Early on we decided to build our own workflow solution as the integration points to our own back-end systems tend to be unique to the BBC. We also decided to use, as much as possible, the same infrastructure and expertise that powers the BBC iPlayer, News, and Sport sites.

With iPlayer, News and Sport we already had services that were very good at Video on Demand (VOD) and live simulcast. What was new for the Olympics was, as Matthew Clark has blogged, the scale of our ambition: 24 simultaneous live streams, thousands of events and more than 2400 hours of content over the two weeks.

Plus we were planning to double or triple any previous streaming peak.

We needed something that could handle the load, allow users to jump to any point in any event stream, and look great on a range of devices.

Simplified diagram of the live streaming architecture

So where we’ve innovated is in our use of HTTP adaptive bitrate streaming. 

ABR was tested in the Wimbledon HD HTTP streaming trial (interesting detail in blog post by Andy Armstrong) that laid the groundwork for the Olympics video.

Delivering live video using HTTP – the same way web pages are delivered to your browser - gives us much greater distribution capacity as it uses existing caches and standard HTTP infrastructure. This allows users to receive high quality streaming video even when we’re delivering the peaks around the big Team GB medal moments.

To get very technical,we decided to use eight h.264 profiles ranging from 54 kbps at 224x126 to 3500 kbps at1280x720. We serve these in several different Adaptive Bit Rate sets targeted at different screen resolutions, device capability and network throughput. We are using two flavours of chunked HTTP delivery: Apple HLS and Adobe HDS. We are still delivering RTMP – an older streaming protocol - for some legacy devices.

A flagship feature of the BBC online Olympics is the interactive video player (IVP, blog post by Alex Perry)  with chapter points that allow a user to jump back to key moments in a live video stream. The streaming technology behind this is HTTP Dynamic Streaming from Adobe with timecodes inserted to allow the player to accurately seek a specific second in the stream.

Oliver Bartlett has blogged about how the Olympic Data services supply the chapter markers and other metadata to the IVP.

Catch-Up

As soon as an event is over the clock starts ticking for us to make the content available as catchup.

The live Flash Media Servers write the HTTP chunks for the live events to disk.

Another first for us is that rather than producing a separate encoding of the same media for the catchup version we recombine the live HTTP chunks back into a single mp4 file and deliver that to the origin of the Content Distribution Network to use for on-demand streaming.

Managing the Streams

To manage so many simultaneous live events we’ve built a live stream management tool we call Marvin.

Marvin maintains all the encoders and all the streaming settings through a single interface. This means that with a single console the AV producers can start and stop encoders, connect the sources, bring graphic slates up and down, route the encoded streams to the correct end points and monitor the status of the encoders.

Marvin also sends the messages to our data services to turn on and off links on the pages and create the right connections, and sends the accurate start and end times to the Olympic data services.

To manage the full and dynamic schedule we’ve built an event scheduling tool that sets up webcasts for each event with scheduled start and end times. This tool allows the same AV producers to manage the full schedule for each day.

A really fantastic aspect of working on the Olympics online video service is seeing all the BBC Future Media teams pulling together to meet the challenge and the support we received from the wider streaming media industry.

If you have any questions on how we delivered the video service I'd love to hear from you.

Marina Kalkanis is Head of Core Services in BBC Future Media, Programmes and On-Demand

So how have we done it?

To start with we've split iPlayer into a front end application written in PHP and a bunch of back end services. Our PHP layer is designed to be infinitely horizontally scalable. We don't maintain state or persist data through the PHP layer. This allows us to easily scale out the iPlayer application onto new servers and into new data centres as our traffic grows.

On the backend we use a combination of Java and Perl. The programme data publishing services have evolved from those we used for iPlayer v2. This is a Perl web service engine we call Dynamite, written using DBIx::Class and Catalyst. It provides a large range of endpoints allowing application developers to pick the feed best suited for their device. We return XML or JSON, structured as iPlayer Object Notation (ION). ION is used not just by PC iPlayer, but across the BBC on any website needing up to date programme metadata. ION and its underlying data structures rely on a constantly evolving and reactive caching strategy.

The social networking services are developed to the OpenSocial 0.8.1 RESTful API specification and built on the Shindig code-base.

The new programme play and favourites tracking service which we call FUSE is a Java web service using Spring and Hibernate. We chose to stick with mySQL for the user activity database because this data is shared between FUSE and Dynamite.

Search and recommendations are also split out as separate services. This allows us to continually refine and improve these with minimal or no impact on service clients. It also allows us to use the features we've developed for iPlayer across other areas of bbc.co.uk.

Finally there is the issue of scaling and performance. We aim to serve all the iPlayer pages even in our peak traffic periods in under two seconds with the majority of requests served in under one second. Our site needs to be available 24 hours a day, 365 days a year. We minimise downtime by ensuring our site can remain up and active even when we lose 50% of our capacity. We implement safety modes in the service that switch off low priority activity when the service comes under load. Non-essential activities are handled asynchronously by message queues. Using Memcached we cache serialised objects and collections ready for quick reads.

We have only just started on the journey of personalising iPlayer and have lots of cool features coming along. Look out for personalised features appearing on the other iPlayer platforms including tablets, TVs, smart Phones and gaming consoles.

Marina Kalkanis is Senior Technical Architect, BBC iPlayer.

Read other recent posts on BBC iPlayer:

• Introducing coloured subtitles on BBC iPlayer
• BBC iPlayer on the iPad update: live streaming of World Cup
• BBC iPlayer press pack for May 2010
• Audience research and BBC iPlayer
• Your feedback on the iPlayer beta
• Going Social with the BBC iPlayer Beta
• Behind the scenes of BBC iPlayer
• BBC iPlayer on the iPad
• BBC iPlayer press pack for April 2010
• Introducing the all new BBC iPlayer (This time it's personal)

This was the look of the BBC iPlayer homepage that we went live with in July of 2007. In those days we were delivering a few hundred TV programmes that could only be viewed by download to Windows PC. Looking back at them now these pages look pretty dull and empty and it was a challenge to get them to change a few times a day.

Now we deliver thousands of TV and Radio programmes. Seems like every day when I get in to work someone has dreamt up a new device to serve iPlayer content to or a new BBC service that wants to contribute programmes into iPlayer. I would never have guessed that the BBC had so many different TV and Radio stations nor that it made such a huge set of programmes.

Today's iPlayer site is built by the iPlayer Publishing Services team. The server side components for today's iPlayer site are built by the iPlayer Publishing Services team.

We are a group of OO perl/Catalyst/mysql developers who are working hard to make the iPlayer pages as rich and engaging as possible while serving 9 million page views a day each in under 2 seconds.

How do we do it?

We use every performance trick we can think of. Our database is flattened, denormalised and heavily indexed to make sql queries as quick as possible. For each query we get we build an abstraction we call our blocklist. These are the lists of programmes on the pages. We serialise these perl objects into memcached ready to reuse for a fixed period of time. We spider our object building to always ensure we have a memcached ready.

iPlayer site traffic gets up to nearly one thousand concurrent requests per second. We share head-end serving with the rest of bbc.co.uk traffic which is handled by 18 Apache web-servers running Apache 2.2 and mod_cache. Behind these sit two Zeus ZXTM load balancing/http cache servers which distribute traffic our eight application servers. Our application server config is Apache2.2/OOperl/Catalyst/memcached. Behind these are 2 mySql DB servers one master and one slave.

In addition to the iPlayer site pages we also serve Atom feeds for all the iPlayer traffic. There are feeds for every programme taste and type of user from CBBC educational TV programmes to Radio Foyle daily schedules.

Things we are working on for next year are ways of making iPlayer more personal, we'll also be refining the way we publish programmes through our player to come up with seamless ways of moving from one programme to the next.

Marina Kalkanis is Senior Technical Architect, BBC Future Media & Technology