Live HD Video Internet Streaming for Real Life First Person Shooter Level 2

I’m on the phone with Peplink partners and resellers all the time – helping with the design and support of some great and frequently challenging projects all over the world.

In any given day I might be supporting an emergency services mobile cellular deployment in Brazil, a bank M2M deployment for ATMs in the US, a global multi-store retail cellular failover network, or a festival WiFi deployment in the UK.  The work is varied but ultimately desk based using all the really useful remote support tools built right into our devices.

I Love Unusual Projects

Occasionally though an end user gets hold of me through linkedin or via word of mouth and asks for something a little bit nuts – and it’s these projects that really add that spark to what I do.

That’s what happened a couple of weeks ago when Dave Reynolds the director at Realm Pictures called to ask for some help. They needed to stream HD video from a disused power station for a live streaming event – a fairly typical Peplink deployment scenario. I was about to hand it off to one of our UK partners who does a lot of live HD video streaming when it got a little more interesting.

They were about to start filming their Real Life First Person Shooter – Level 2 and needed to stream from a gopro strapped to an actor’s head as he runs about deep in the bowels of the power station, whilst also interacting with the video stream of the remote viewer and they needed to do the install in 2 days time.

The need for bi directional video streaming – effectively a real time HD video call, meant that using satellite broadband connectivity would be undesirable due to its high latency, so cellular data connectivity (if fixed lines were unavailable) was the target transport medium.

As you’ll know if you’ve ever used skype on your smartphone on the go, video streaming over cellular can be fantastic, or it can be absolutely useless. The result is 100% dependant on mobile operator coverage and attainable bandwidth (in both directions for two way video) and the usability is heavily influenced by latency. Two days was too tight a timescale for a typical approach of onsite cellular survey by a Peplink partner to ascertain what coverage was available – not to mention all the additional excitement of wifi network infrastructure deployment in a building as complicated, hostile to RF, or as large as a disused power station.

No, if it was going to work, I’d need to call in some favours and get some help and go to site myself to make sure. I needed a peplink partner who had done a lot cellular deployments and worked with Ubiquiti Wireless kit before (as that is what the production company had a stock of) who wouldn’t mind the slightly crazy timescales. So I called the guys at Grapevine Connect who were perfect as they not only know what they are doing with both Peplink and Ubiquiti hardware but they are also just down the road from me. Two days later I was in a Grapevine van with two of their senior guys Martin and Keith on my way to the location with the cargo area stuffed full of Peplink kit, drums of data cable, tool kits and enough cable ties, velcro and duct tape to deal with just about any challenge that might come our way.

The Location

The location was Fawley an oil-fired power station, on the shores of Southampton Water in Hampshire. It was built in the 1960’s to turn the residual oil waste product from a neighbouring oil refinery into electricity. When running it could generate 1070MW of electricity – enough energy for 1 million people (the working population of Hampshire) but it was shut down in 2013.

The site was massive, on a scale I have never seen before. The main turbine room being long enough that you could stand at one end and not quite see the other through the pipes and turbines. My first thought on seeing it was that we didn’t have enough data cable with us. In fact, you could have filled a lorry full of boxes of CAT6 and still not have enough. Thankfully we discovered that the production company were limiting themselves to one end of the site that included the insanely cool circular control room (complete with banks and banks of dials and switches) and the end of the vast turbine hall that were linked via a flybridge. Even so this was  about 20-30,000 sqm of area which still felt pretty daunting.

Fixed Line Broadband- Sadly a Dead End

The media company were hopeful that there might be a fixed line broadband service active on site, so we poked our noses into the comms room off of the air bridge that links the control room to the turbine hall armed with line test kit and optimism. Sadly although we had a a couple of false positives which gave us hope, eventually we realised that there were no services we could access so we had to rely on cellular connectivity as the only way forward.

The Cellular Survey

The first and most important job was to do a full cellular survey of the site to find out what cellular coverage there was and which operators were available. At this stage we still didn’t know if we would be able to supply them with enough bandwidth for their project – since the power station was naturally geographically isolated by design. Grapevine had brought a couple of their cloudcases along (BR1s in portable rugged cases with inbuilt antennas) which were perfect for this as we could roam the area with the cases powered by battery packs and run speed tests using sim cards from different providers from as many places as possible.

The main ‘stage’ for the event was the turbine hall from which they were basing their IT infrastructure in the old glass fronted foremans office – so we started there first. Signal strength suffered horrendously inside of the steel and glass building as you would expect, with a maximum throughput obtained in the region of 0.3Mbps down and 0.1-0.2Mbps Up, Nowhere near enough bandwidth for anything.

So we moved slowly outwards still in the main hall but towards the large full height industrial doors hoping that we might get lucky and see the benefit of cellular RF reflections – sadly not, it soon became clear that we would need our antennas to be outside of the structure and that our only hope was to try and get line of sight to the cell towers on the east coast of the southampton waters.

Still at ground level we slowly worked our way further and further east towards Southampton Water, and out of the RF ‘shade’ of the circular control tower we started to see better throughput in the region of 5Mbps down and 7Mbps up and as we moved even further away from the control tower speeds dramatically improved, at one point we got 15Mbps down and 24Mbps up which would have been perfect however completely impractical.

Location, Location, Location

The scale of the place is hard to describe fully, but to get the highest throughput possible we needed to be over 200m away from the foremans office where the connectivity was needed, and over 50m away from the control tower with the sweet spot located almost in the middle of a car parking area with no mains power and no physical security or protection for the gear – so we needed to stop and rethink the deployment.

The only conceivable place that we thought a reasonable signal strength was achievable was in the circular control tower since that faced the sea, however earlier testing in the control room had been disappointing (1Mbps down, 0.8Mbps Up) which didn’t match with our more positive readings at ground level – so more investigation was required.

Once we were back up in the control tower measured cellular throughput was still very low. We noticed that the windows were covered in a slightly reflective silver sunglare protection and suspected that this was acting as a ground plane so worked our way round the room trying to find a window that opened to test further.

With the window open and antennas in the breeze we were getting exceptional throughput on both Three and Vodafone – between 17-19Mbps down and 21-26Mbps up! It had taken 6hrs to get to this point and we had a lot of work left to do but our bandwidth issues were finally under control.

Vodafone

Three

Cable Routes in Power Stations are Awful

We had to somehow get an ethernet link from the 3rd floor of the control tower where the cellular routers were located back to the 1st floor glass fronted foremans control centre deep within the turbine hall. Lacking an easy direct cable route meant that we were looking at somewhere between 160-210m run of cable that needed to pass through self closing fire doors and down stairwells – a painful run even if it worked but also one likely to get damaged by the heavy fire doors.

We knew there had to be an existing comms link between the control room and the foremans office of some kind (most likely fiber) and although we spent some time looking for one we couldn’t find it since nothing was labelled or presented in an obvious fashion at either location.

So we ended up doing the only approach that had a chance of working – running data cable from the cellular routers along the corridor in the air bridge back to a gantry high up in the turbine hall and then deploying a point to point link between the gantry and the foremans office.

Time was really against us at this point, we had till 2am when we would be thrown out of the station by security, but it was a 3hr drive back home and we all had work in the morning, so fast and efficient became the driving forces.

The Deployment

In the end the deployment looked like this:

• A BR1 using Three (LTE) acting as a remote management entry point to the network
• A HD2 with Mediafast using two Vodafone LTE cellular connections and SpeedFusion Bonding providing connectivity for the HD video Stream.
  This created a SpeedFusion VPN back to a hosted instance of Fusionhub our SpeedFusion VPN concentrator.
• A Balance One Core acting as the default gateway for the on site data network with the HD2 on its WAN1 and the BR1 on its WAN2. This was located as close as possible to the gantry in the turbine hall at the very end of the airbridge.
• A transparent P2P WiFi bridge from Ubiquiti (Nanostation M2’s that Grapevine had brought with them) that linked the gantry to the the balcony at the back of the foremans office.
• In the foremans office was a 24port unmanaged switch for local connectivity.

We decided to deploy both the BR1 and the HD2 for a couple of reasons, the first was that since live video is so latency sensitive we didn’t want any remote management traffic that we might generate as we remotely fine-tuned the network to go anywhere near the video stream when they were broadcasting.

The second was that since the BR1 was on Three and the HD2 was using Vodafone (which had better throughput availability on average), we could use the BR1 as a failover path if we needed to, or change the config and include it in the bonded VPN session if we needed to as well.

In Summary

I’m really proud of what Grapevine and I achieved for Realm Pictures. With two days notice, and within a short period of time on site, we managed to deliver a reliable high bandwidth bonded cellular internet connectivity for a high profile time sensitive event in a hostile environment – an achievement only possible due to the quality of the Peplink gear we were using combined with our stubborn determination.

Dave Reynolds was really impressed with the connectivity, he said:
‘we didn’t have a single problem with the cellular – it was super robust, very impressive! [and] on a shoot like this [that is] utterly utterly vital.’

You can view the entire Level Two Video here
And the behind the scenes video is available here