E07: The Video Insiders talk with a pioneering software development company who is at the center of the microservices trend in modern video workflows. Featuring Dom Robinson & Adrian Roe from id3as.
Beamr blog: https://blog.beamr.com/2019/02/04/microservices-good-on-a-bad-day-podcast/
Following is an undedited transcript of the episode. Enjoy, but please look past mistakes. Mark & Dror
Intro: 00:00 The Video Insiders is the show that makes sense of all that is happening in the world of online video as seen through the eyes of a second generation Kodak nerd and a marketing guy who knows what I-frames and Macroblocks are. And, here are your hosts, Mark Donnigan and Dror Gill.
Mark Donnigan: 00:22 Well, welcome back to the Video Insiders. It's so great to be here. Dror, how are you doing?
Dror Gill: 00:29 I'm doing great and I'm really excited to do another episode of the Video Insiders. I would say this is probably the best part of my day now doing the Podcast. Although, watching video all day isn't bad at all.
Mark Donnigan: 00:45 That's not a bad job. I mean, hey, what do you tell your kids?
Dror Gill: 00:49 So, exactly, this is [crosstalk 00:00:52]. I work part-time out of my home office and my daughter comes in after school and she sees me watching those videos and she says, "Dad, what are you doing?" So, I said, I'm watching videos, it's part of my work. I'm checking quality, stuff like that. Then she says, "What? That's your work? You mean they pay you to do that? Where can I get a job like that? You get paid to watch TV."
Dror Gill: 01:18 Now, of course, I'm not laid back on a sofa with some popcorn watching a full length movie, no. I'm watching the same boring video clip again and again, the same 20, 30 seconds segments, and I'm watching it with our player tool, with Beamr view and typically one half is flipped over like butterfly mode. And then, you're pausing on a frame and you're looking for these tiny differences in artifacts. So, it's not exactly like watching TV in the evening, but you get to see stuff, you get to watch content, it's nice but could get tiring after a while. But, I don't think I'll ever get tired of this Podcast Mark.
Mark Donnigan: 02:04 No, no. I know I won't. And, I think going back to what you do in your day job watching video, I think our listeners can relate to. It's a little bit of a curse, because here you are on a Friday night, you want to relax, you just want to enjoy the movie, and what do you see? All of the freaking artifacts and all the ... And, you're thinking that ABR recipe sure could have been better because I can see it just switched and it shouldn't have, anyway, I think we can all relate to that. Enough about us, let's launch into this episode, and I know that we're both super excited. I was thinking about the intro here, and one of the challenges is all of our guests are awesome, and yet it feels like each guest is like this is the best yet.
Dror Gill: 02:56 Yeah. Really today we have two of really the leading experts on video delivery. I've been running into these guys at various industry events and conferences, they also organize conferences and moderate panels and chair sessions, and really lead the industry over the top delivery and CDNs and all of that. So, it's a real pleasure for me to welcome to today's Podcast Dom and Adrian from id3as, hi there?
Adrian Roe: 03:26 Hey, thank you very much.
Dom Robinson: 03:27 Hey guys.
Adrian Roe: 03:27 It's great to be on.
Dom Robinson: 03:28 How are you doing?
Dror Gill: 03:29 Okay. So, can you tell us a little bit about id3as and stuff you do there?
Adrian Roe: 03:34 Sure. So, id3as is a specialist media workflow creation company. We build large scale media systems almost always dealing with live video, so live events, be that sporting events or financial service type announcements, and we specialize in doing so on a very, very large scale and with extremely high service levels. And, both of those I guess are really crucial in a live arena. You only get one shot at doing a live announcement of any sort, so if you missed the goal because the stream was temporary glitch to that point, that's something that's pretty hard to recover from.
Adrian Roe: 04:14 We've passionate about the climate and how that can help you build some interesting workflows and deliver some interesting levels of scale and we're primary constructors. Yeah, we're a software company first and foremost, a couple of the founders have a software background. Dom is one of the original streamers ever, so Dom knows everything there is to know about streaming and the rest of us hang on his coattails, but have some of the skills to turn that into one's a note, so work for our customers.
Dror Gill: 04:46 Really Dom, so how far back do you go in your streaming history?
Dom Robinson: 04:50 Well, anecdotally I sometimes like to count myself in the second or third webcasters in Europe. And interestingly, actually one of the people who's slightly ahead of me in the queue is Steve Clee who works with you guys. So, did the dance around Steve Clee in the mid '90s. So, yeah, it's a good 20, 23 years now I've been streaming [inaudible 00:05:12].
Dror Gill: 05:11 Actually, I mean, we've come a long way and probably we'll talk a bit about this in today's episode. But first, there's something that really puzzles me is your tagline. The tagline of id3as is, good on a bad day. So, can you tell us a bit more about this? What do you mean by good on a bad day?
Adrian Roe: 05:33 We think is probably the most important single facet about how your systems behave, especially again in a live context. There are hundreds or possibly even thousands of companies out there who can do perfectly good A to B video encoding and transcoding and delivery when they're running in the lab. And, there's some great tools, open source tools to enable you to do that, things like FFmpeg and so on. What differentiates a great service from a merely good service though is what happens when things go wrong. And especially when you're working at scale, we think it's really important to embrace the fact that things will go wrong. If you have a thousand servers running in your x hundred events at any one particular time, every now and then, yeah, one of those servers is going to go up in a puff of smoke. Your network's going to fail, or a power supply is going to blow up, or whatever else it may be.
Adrian Roe: 06:31 And so, what we think differentiates a great service from a merely good one is how well it behaves when things are going wrong or ranji, and partly because of the technology we use and partly because of the background we come from. Technically, when we entered the media space, so as a company that was about eight years ago, obviously Dom's been in the space forever, but as a company it's been eight years or so, we came to it from exactly that angle of how can we ... So, our first customer was Nasdaq delivering financial announcements on a purely cloud based system, and they needed to be able to deliver SLAS to their customers that were vastly higher than the SLAS you could get for any one particular cloud service or cloud server. And so, how you can deliver a fantastic end to end user experience even when things inside your infrastructure are going wrong, we think is much more important than merely, can you do an A to B media chain?
Mark Donnigan: 07:27 That's interesting Adrian. I know you guys are really focused on micro services, and maybe you can comment about what you've built and why you're so vested in data architecture.
Adrian Roe: 07:39 With both things, there's nothing new in technology. So, Microservices as a phrase, I guess has been particularly hot the last, I don't know, three, four years.
Mark Donnigan: 07:49 Oh, it's the buzzy, it's the buzzy word. Dror loves buzzy words.
Dror Gill: 07:54 Micro services, buzz, buzz.
Mark Donnigan: 07:54 There we go. I'm afraid you have to hear the rap, you have to hear his rap. I'm telling you it's going to be number one on the radio, number one on the charts. It's going to be a hit, it's going to be viral, it's going to be [inaudible 00:08:08].
Adrian Roe: 08:09 So, our approach to Microservices I'm afraid is grounded in the 1980s, so if we're going to do a rap at that point, I'd need to have a big bouffant hair or something in order to do my Microservices-
Mark Donnigan: 08:18 And new eyes.
Dom Robinson: 08:21 You left your flares in my house dude.
Adrian Roe: 08:23 Oh, no, my spare pairs are on, it's okay. Actually, a lot of that thinking comes from the Telco space where when we were starting to get into ... In a past life I used to build online banks and big scale systems like that, but one of the things that was interesting when we came to media is actually if you've got 500 live events running, that's a big system. The amount of data flowing through that with all the different bit rates and so on and so forth is extremely high. Those 500 events might be running on a thousand servers plus in order to give you a full scale redundancy and so on and so forth, and those servers might well be spread across three, four, five different data centers in three, four, five different continents.
Adrian Roe: 09:14 And, there are some properly difficult problems to solve in the wider space rather than specifically in the narrow space of a particular single element to that workflow. And, we did some research a while back, we said actually other people must have faced some of these challenges before. And, in particular the Telco space has faced some of these challenges for a long time, and people get so used to just being able to pick up the phone and have the call go from A to B, and the technology by and large works so well that you don't really notice it's there, which is actually another good strap line I think, technology is so good you ignore it, that's what we aspire to.
Adrian Roe: 09:51 So, we came across a technology called Erlang, which takes a whole approach to how you build systems. It's different to the traditional. As I say, in itself is not a new technology and that's one of the things we like about it, but basically it says the problems that Erlang was trying to solve when it was created back in the '80s was specifically for things like mobile phones, which is where you would have a mobile phone switch, would be a whole bunch of proprietary boards, each of which could handle maybe, I don't know, five or 10 calls or something, and they'd be stuck together and a dish great big rack with some kind of backplane joining them altogether. And, the boards themselves were not very reliable, and in order for the mobile or for the Telcos to be able to deliver a reliable service using this kind of infrastructure, if any one particular board blew up, the service itself had to continue and other calls, it was really important to those other calls weren't impacted and so on and so forth.
Adrian Roe: 10:48 So, this language Erlang was invented specifically to try and solve that class of problem. Now, what was interesting is if you then wind the clock forward 20, 30 years from that particular point and you consider something like the cloud, the cloud is lots and lots of individual computers that on their own aren't particularly powerful and on their own aren't particularly reliable, but they're probably connected together with some kind of LAN or WAN that actually is in pretty good shape.
Adrian Roe: 11:17 And, the challenges that back then were really customed to the mobile and network space suddenly become incredibly good patterns of behavior for how you can build high scale cloud systems and super reliable cloud systems. And so, this as is always the case, these new shiny technologies, Erlang, for example, had its moment in the sun about a year or so back when WhatsApp was bought by Facebook, because when WhatsApp were bought by Facebook for $18,000,000,000 or whatever it was, I believe that WhatsApp had a total of 30 technical staff of which only 10 were developers, and they build all of their systems on top of Erlang and got some major advantage from that.
Adrian Roe: 11:57 And so, when we came into the whole media space, we thought that there were some very interesting opportunities that would be presented by adopting those kinds of strategies. And now, what's nice then about what a Microservices come into that, so in Erlang or the way we build systems, you have lots of single responsibility, small bits of function, and you gather those bits of function together to make bigger, more complex bits of function and then you gather those together to make progressively more larger scale and more complex workflows. And, what's really nice about that as a strategy so people are increasingly comfortable with using Microservices where I'll have this to do my packaging and this to do my encoding, and then I'll plug these together and so on and so forth.
Adrian Roe: 12:46 But, when your language itself is built in those kinds of terms, it gives you a very consistent way of describing about the user experience all the way through your stack. And, the sorts of strategies you have for dealing with challenges or problems that are very low level are exactly the same as the strategies you have for dealing with server outages, and so on and so forth. So, it gives you a very consistent way that you can think about the kind of problems you're trying to solve and how to go about them.
Dror Gill: 13:10 Yeah, that's really fascinating. So basically, we're talking about building a very reliable system out of components where not all of these components are reliable all the time, and inside those components are made out of further sub components, which may fail.
Adrian Roe: 13:28 Correct, yeah.
Dror Gill: 13:29 And then, when you employ a strategy of handling those failures and failing over to different components, you can apply that strategy at all levels of your system from the very small components to the large servers that do large chunks of work.
Adrian Roe: 13:45 I could not have put it better myself, that is exactly right. And, you get some secondary benefits, so one is I am strongly of the opinion that when you have systems as large and as complex as the media workflows that we all deal in, there will be issues. Things will go wrong either because of physical infrastructure role, just because of the straight complexity of the kinds of challenges you're looking to meet. So, Erlang would take an approach that says let's treat errors as a first class citizen, let's not try and pretend they're never going to happen, but let's instead have a very, very clear pattern of behavior about how you go about dealing with them, so you can deal with them in a very systematic way. And, if those errors that are very, very micro level, then the system will probably replace the things that's gone bad, and do so in a few well under a fractions of a millisecond. So, you literally don't notice.
Adrian Roe: 14:41 We had one particular customer where they had a component that allowed them to patch audio into a live media workflow, and they upgraded their end of that particular system without telling us or going through a test cycle or something which was kind of disappointing. And, a week or so after their upgrade, we were looking at just some logs from an event somewhere, and they seemed a bit noisier than usual. We couldn't work out why and the event had been perfect, nothing had gone wrong, and we discovered that they started to send us messages, one part of the protocol, so they were just incorrectly sending us messages as part of this audio integration that they'd done and they were just sending us junk.
Adrian Roe: 15:24 And, the handler forwarded our end was doing what it ought to do in those particular cases that was crashing and getting itself replaced. But, because we designed the system really well, the handler and the logic for it got replaced. The actual underlying TCP connection, for example, stayed up and there wasn't a problem. And, actually we're having to restart the handler several times a second on a live two way audio connection and you literally couldn't hear that it was happening.
Mark Donnigan: 15:49 Wow.
Adrian Roe: 15:49 Yeah. So yeah, you can get ... But, what's nice is exactly the same strategy in the way of thinking about things and works. Yeah, right at the other level where I've gone seven data centers, and 1000, or 1500 servers running and so on and so forth, and it gives you a camera and a consistent strategy for how you reason about how you're going to behave in order to deliver a service that just keeps on running and running and running even when things go bad. I will give one example, then I'll probably let Dom share some of his views for a second, which was there was a reasonably famous incident a few years back when Amazon in US East just disappeared off the map for about four days and a number of very large companies had some really big challenges with that, and frankly we were just offline for four days.
Adrian Roe: 16:36 We had 168 servers running in US East at the time for Nasdaq, one of our customers, we did not get a support call. And so, all of the events that were running on there failed over to other servers that we're running in US West typically. About five minutes later we were back in a fully resilient setup, because we'd created infrastructure in Tokyo and Dublin and various other data center, so that had US West disappeared off the face of the earth as well. Again, we might've got a support call the second time around, but we literally read about it in the papers the next day.
Mark Donnigan: 17:06 That's pretty incredible. Are there any other video systems platforms that are architected on Erlang, or are you guys the only ones?
Adrian Roe: 17:15 The only other one I am aware of out of the box is a company that specializes more in the CDN and final content delivery side of things, so we're not quite unique, but we are certainly highly, highly unusual.
Mark Donnigan: 17:28 Yeah. Yeah. I didn't want to go to Dom, and Dom with your experience in the industry, I'm curious what you're seeing in terms of how companies are architecting their workflows. Are you getting involved in, I guess evolutionary projects, that is you're extending existing platforms and you're in some cases probably shoe honing, legacy approaches, solutions, technologies, et cetera, to try and maybe bring them to the cloud or provide some sort of scale or redundancy that they need? Or, are people just re architecting and building from the ground up? What are people doing out there and what are specifically your clients doing in terms of-
Dom Robinson: 18:20 So, it's interesting, I was talking, I did a big review of the Microservices space for Streaming Media Magazine, which came out I think in the October edition this year, and that generated quite a lot of conversations and panel sessions and so on. When we've been approached by broadcasters who have established working workflows, and they're sometimes quite testy because they've spent a lot of time and then they're emotionally quite invested in what they might have spent a decade building and so on. So, they often come with quite testy challenges, what advantages would this bring me? And quite often, there's very little advantage in just making the change for the sake of making the change. The value really comes when you're trying to scale up or take benefit from scaling down. So, with a lot of our financial needs clients the cycle of webcasts, if you'd like a strongly quarterly though, it's all about financial reporting at the end of financial quarters. So, they often want to scale down their infrastructure while during the quiet weeks or quiet months because it saves them costs.
Dom Robinson: 19:25 Now, if you're doing 24/7 linear broadcasting, the opportunity to scale down may simply never present itself, you just don't have the opportunity to scale down. Scaling up is a different question, but scaling down, if it's 24/7, there's no real advantage to scaling down, and this is true of cloud as much as it is of Microservices specifically. But, when people come to us and say, right, we've really want to make that migration, they sometimes start with the premise that they'd like to take tiny little pieces of the workflow, and just migrate those little tiny incremental steps. In some cases we may do that, but we tend to try to convince them to actually build a Microservice architecture or virtualized architecture to run in parallel. So, quite often we might start with the client by proposing that they look at their virtualized strategies disaster recovery strategy in the first instance. And then, what happens is after the first disaster, they never go back to their old infrastructure.
Mark Donnigan: 20:21 I'm sure, yeah.
Dom Robinson: 20:22 And after that, they suddenly see they have all the benefits and it is reliable and despite the fact that they have no idea where on earth this physically is happening, it's working and it works really reliably. And, when it goes wrong, they can conjure up another one in a matter of seconds or minutes. These are not apparent until the broadcaster actually puts them into use. I spent 20 years trying to convince the broadcast industry that IP was going to be a thing, and then overnight they suddenly embraced it fully, and these things people do have epiphany's and they suddenly understand the value.
Dom Robinson: 20:56 Disaster recovery has been a nice way to make people feel comfortable because it's not a suggestion of one day we're going to turn off your trusted familiar, nailed down tin and move it all into something you have no idea where it is, what it's running on, how it's running and so on. People are risk averse naturally in taking that type of leap of faith, but once they've done it, they almost invariably see the benefits and so on. So, it's about waiting for the culture in the larger broadcasters to actually place that confidence in the, if you like, the internet era, which generally means as people who are being cynical. I used to make testy comments on panel sessions about the over '50s, '60s, I don't know where you want to put your peg in there. Once those guys finally let internet natives take control, that's when the migration happens.
Mark Donnigan: 21:48 Yeah, that's interesting. I can remember going back, oh, 10 years or more and sitting in the cable show which no longer exists, but certain sessions there and Cisco was presenting virtualized network function. And, when the room would always be packed and you'd have a sense if you're sitting in these sessions like this is really happening. This is, wow, this is really happening in all the biggest MSLs were there, all the people were there, right? And then, you'd come back the next year, it'd be the same talk the same people in the room, then come back the next year after that and nobody was [crosstalk 00:22:25], because it's the future.
Dom Robinson: 22:23 Yeah, absolutely.
Dror Gill: 22:28 It was always the future I was making fun of.
Mark Donnigan: 22:30 Now, the switch has absolutely flipped and we're seeing that even on the codecs side, because there was a time where unless you were internet native as you said, you needed a full solution, a black box. It had to go on a rack, it had to ... That's what they bought. And so, selling a codec alone was a little bit of a challenge, but now they can't use black boxes, and they're ... So.
Dom Robinson: 22:58 Sometimes I liken it to the era of HI-FI as digital audio and MP3 started to arrive, I was quite involved in MP3 as it emerged in the mid '90s. And, I have over the last two decades flip flop from being the musicians, worst enemy to best friend to worst enemy to best friend, and everybody just depends on the mood of the day. I was reflecting, and this is a bit of a tangent, but I was reflecting when you guys were talking about watching for artifacts in videos. I've spent so long watching 56K blocky video that Adrian, Nick and Steven, the rest of the team never ever let me give any opinion on the quality of video, because I'm quite happy watching a 56K video projected on my wall three meters wide and it doesn't bother me, but I'm sure Dror would be banging his head against the wall if he [inaudible 00:23:47] videos.
Dror Gill: 23:49 No, I also started with 56K video and real video, and all of those the players and still in the '90s, but I managed to upgrade myself to SD and then to HD, and now if it's not HDR, it's difficult to view. But in any case, if we look at this transition that is happening, there are several levels to this transition. I mean, first of all, you make the transition from hardware to software then from the software to the cloud, and then from regular software running in the cloud and VMs to this kind of Microservices architecture with Dockers. And, when I talk to customers they say, yeah, we need it as a Docker, we're going to do everything as a Docker. But then, as Mark said, you're not always sure if they're talking about the far future, the new future, the present, and of course it changes if you're talking to the R&D department or you're talking with the people who are actually doing the day to day production.
Adrian Roe: 24:51 There were some interesting ... And, I think Docker, this maybe a slightly unpopular thing to say, but yeah, so I think Docker is fantastic and yeah, we use it on a daily basis and development and it's a great on my laptop, I can simulate a cluster of eight servers or doing stuff and failing over between them and so on and so forth and it's fantastic. And, and we've had Docker based solutions in production for four years, five years, certainly a long time, and actually we were starting to move away from Docker as a delivery platform.
Dror Gill: 25:22 Really? That's interesting. So, you were in the post Docker era?
Adrian Roe: 25:26 Yes, I think just as other people are getting very excited that their software can run on Docker, which I always get confused with announcements like that, because Docker is essentially another layer of virtualization, and strangely enough people first all got excited because their software would run not on a machine but on a virtual machine and it takes quite a strange software requirement before the software can really even tell the difference between those. And then, you move from a virtual machine to a Docker type environment.
Adrian Roe: 25:52 Yeah. Docker of course being conceptually nothing new and yeah, it's a wrapper around something the Linux kernel has been able to do for 10 years or so. Yeah. And, it gives you certain guarantees about kerniless and that the sandbox isn't going to interfere with the sandbox and so on and so forth. And, if those things are useful to you, then absolutely use Docker to solve those business problems.
Adrian Roe: 26:13 And another thing that Docker can do that again solves a business problem for me when I'm developing is I can spin up a machine, I can instantiate a whole bunch of stuff, I can create virtual networks between them, and then when I rip it all down my laptop's back in pretty much the same state as it was before I started, and I have some guarantees around that. But especially in a cloud environment where I've got a premium job coming in of some sort, I'll spin up a server to do that and probably two servers in different locations to be able to do that. And, they'll do whatever they need to do and yeah, there'll be some complex network flows and so on and so on and so forth to deliver that.
Adrian Roe: 26:48 And then, when that event's finished, what I do is I throw that server in the bin. And so, actually Docker there typically is just adding an extra abstraction layer, and that abstraction layer comes at a cost in particular incidence of disk I/O and network I/O that for high quality video workflows you want to go into with your eyes open. And so, when it's solving a business problem for you, I think Docker is a fantastic technology, and some very clever people are involved and so on and so forth. I think there's a massive amount of koolaid been drunk just to see if Docker where it's actually adding complexity and essentially no value.
Dror Gill: 27:25 So, I would say that if you have, as I said, if you have a business problem, for example, you have Linux and Windows servers, it's a given you can't change that infrastructure and then you want to deploy a certain task with certain servers, and you wanted to work across them seamlessly with those standard interfaces that you mentioned, then Docker could be a good solution. On the other hand, what you're saying is that if I know that my cluster is fully Linux, certain version of Ubuntu, whatever, and because that's how I set it up, there's no advantage in using the Dockers because I can plan the workflow or the workload on each one of those servers, and at the level of cloud instances launch and terminate them, and then I don't need the Docker. And the issue of overhead, we haven't seen a very large overhead for Docker, we always compare it to running natively. However, we did find that if your software is structured in a certain way, it can increase the overhead of Docker beyond the average.
Dom Robinson: 28:31 Something important that came up in some of the panels, Streaming Media West and content delivery world recently on this topic, at the moment people talk synonymously about Microservices and Docker, and that's not true. Just because something's running in Docker does not mean you're running a Microservices architecture. In fact if you dig under the ... All too often-
Dror Gill: 28:50 Right, it could be a huge one of the thick servers. Servers that are just running on Docker.
Dom Robinson: 28:54 Exactly. All too often people have just simply drop their monolith into a Docker container and called it a Microservice, and that's a ... Well, I won't say it on your Podcast, but that's not true. And, I think that's very important, hence we very much describe our own Erlang based architecture as a Microservices architecture. Docker is as Adrian was explaining, it's nice to have in certain circumstances, it's an essential, but in other circumstances it's just not relevant to us. So, it is important that Docker is a type of virtualization and is nothing to do with Microservices architecture, and it's a very different thing. So, well Adrian might kick me under the virtual table.
Adrian Roe: 29:27 No, no, that's all ... Yeah, there's a lot of people who will say if you take an application and you turn it into ... You take a monolithic application and Microservicize it what you have is a monolithic application that's now distributed. So, you've taken a hard problem and made it slightly harder.
Dom Robinson: 29:44 Exactly.
Adrian Roe: 29:45 So, what's probably more important is that you have good tools and skills and understanding to deal with the kinds of challenges you get in distributed environments. And, actually understanding your own limitations is interesting there. I think if you look at how one coordinate stuff within a particular OS application, then Microservices are a great way of structuring individual applications, and they can cooperate, and they're all in the same space, and you can replace bits of them and that's cool. And then, if you look at one particular server, again, you're Microservices architecture there might go, okay, this component is an an unhealthy state, I'm going to replace it with a clean version and yeah, you can do that in very, very quick time and that's all fantastic.
Adrian Roe: 30:33 And then, maybe even if I'm running in some kind of local cluster, I can make similar decisions, but as soon as I'm running in some kind of local cluster, you have to ask the question, what happens if the network fails? What's the probability of the network failing? And if it does, what impact is that going to have on my service? Because yeah, it's just as bad typically to have two servers trying to deliver the same instance of the same live services as it is to have none, because there'll probably be a closed network floods and all sorts of bad things can happen as a result, so.
Adrian Roe: 31:08 And then, if you look at a system that's distributed over more than one day center that absolutely just going, oh, I can't see that other service. Yeah, so Microservice is part of my overall delivery. Making decisions based on that is is something you need to do extremely carefully and there's an awful lot of academic work done around consensus algorithms in the presence of network splits and so on and so forth, and it's not until you understand the problem quite well that you actually understand how damned hard the problem is. You're just naive understanding of it is, oh, how hard can it be just to have three servers agree on which of them should currently be doing x, y, z job? Turns out it's really, really, really hard, and that you stand on the shoulders of giants because there's some amazing work done by the academic community over the last few decades, go and leverage the kind of solutions that they've put together to help facilitate that.
Dom Robinson: 31:59 I think one of the upsides of Docker though is it has subtly changed how dev teams are thinking, and I think it's because it represents the ability to build these isolated processes and think about passing data between processes rather than just sharing data in a way a monolith might have done. I think that started people to architect in a Microservices architecture. I think people think that that's a Docker thing, but it's not. Docker is more of a catalyst to it than actually bringing about the Microservices architecture.
Mark Donnigan: 32:33 That's interesting Dom. I was literally just about to make the point or ask the question even. I wonder if Docker is the first step towards truly Microservices architecture for a lot of these organizations, and I think Adrian did a great job of breaking down the fact that a lot of maybe what is getting sold or assumed to be Microservices really isn't, but in reality it's kind of that next step towards a Microservices architecture. And, it sounds like you agree with that.
Dom Robinson: 33:09 Yeah, yeah, yeah. I think it's part of the path, but it's a-
Mark Donnigan: 33:12 That's right.
Dom Robinson: 33:13 Going back to my original statement Doc-
Adrian Roe: 33:13 I am not even sure that strongly it's an available tool in this space.
Mark Donnigan: 33:18 It's an available tool, yeah.
Adrian Roe: 33:18 You can absolutely build Microservices at dentonville Docker anywhere. Yeah.
Mark Donnigan: 33:24 Sure. Absolutely. Yeah. I wasn't saying that Docker's a part of that, but I'm saying if you come from this completely black box environment where everything's in a rack, it's in a physical location, the leap to a truly Microservices architecture is massive. I mean, it's disruptive on every level.
Adrian Roe: 33:46 And, it's a great tool, it's part of that journey. I completely do agree with that.
Mark Donnigan: 33:48 Yeah, exactly. Exactly. Well, this leads into a conversation or a topic that's really hot in the industry right now, and that's a low latency. I was chuckling, I was walking around Streaming Media West just couple of weeks ago, and I don't think there was one booth, maybe there was one, I just didn't see it. Maybe the Panasonic camera booth, they didn't have low latency plastered all over it, but every booth, low latency, low latency,
Adrian Roe: 34:16 There's some interesting stuff around low latency because there's a beautiful reinvention of the wheel happening because, [crosstalk 00:34:28].
Mark Donnigan: 34:29 Well, let's talk about this because maybe we can pull back a little bit of the, I don't know the myths that are out there right now. And also, I'd like to have a brief real honest conversation about what low latency actually means. I think that's one of the things that, again, everybody's head nods, low latency. Oh yeah, yeah, yeah, yeah. We want that too, but then you're like what does it mean?
Dror Gill: 34:57 Yeah, everybody wants it. Why do they want it, is an interesting question. And, I heard a very interesting theory today because all the time you hear about this effect of if you're watching a soccer game and you have a lot of latency because you're viewing it over the internet and somebody else has cable or satellite and they view it before you, then you hear all those roars of the goal from around the neighborhood and this annoys the viewer.
Dror Gill: 35:25 So, today I heard another theory that that's not the problem of low latency because to block those roars you can just isolate your house and put on headphones or whatever. The real problem that I heard today is that, if there's a large latency between when the game actually happens and when you see it, then you cannot affect the result of the game. Okay? So, the theory goes like this, you're sitting at home, you're wearing your shirt and your fan, and you're sitting in a position that is a lucky position that will help your team. So, if the latency is high then anything you do cannot affect the game because it's too late, because the latency is low you'll have some effect over the result of the game.
Adrian Roe: 36:13 When TiVo was brand new and there was the first personal video digital video recorders were a thing. They had this fantastic advert where somebody was watching an american football game, and as they're in sudden death overtime and the kick is just about to do a 45 yard kick. Yeah, and if it goes over, they win the game and if it doesn't, they lose the game. Kickers just running up towards it and he hits pools on the live stream, runs off to the church, prays for half an hour, comes back, and it's really good.
Dror Gill: 36:47 Oh, so that's the reason for having a high latency.
Adrian Roe: 36:55 It's interesting, the primary businesses in broadcast distribution as In over the air type distribution, but we do a bunch of the hybrid TV services, and as part of that we actually have to do the direct hand off to a bunch of the TVs and set top boxes and so on and so forth. Principally because, the TVs and sets of boxes are so appallingly behaved in terms of the extent to which they deal with then follow standards and so on. So, in order to deliver the streams to a free view plus HDTV in the UK, we just deliver them a broadcast quality transport stream as a progressive download, and entirely so this has been live in the field for, I don't, seven years or something. And entirely without trying to, we have an end to end latency of around two seconds from when the viewer in the home sees it on the TV, as opposed to the original signal coming off the satellite. And nowadays, that would be called super low latency and actually clever and remarkable and so on and so forth. And actually, it's primarily created by the lack of segmentation.
Mark Donnigan: 38:01 That's right.
Adrian Roe: 38:03 What's happened that suddenly made you have an RTMP streams. It's depended a little bit on how much buffering you had in the player and so on, but they typically have an end to end latency in a video workflow based around RTMP, five, six seconds, that was normal and they would really comment on it. And now, suddenly that you have segment oriented distribution mechanisms like HLS and Dash and all these kinds of things, people talk about low latency and suddenly they mean five to 10 seconds and so on and so forth. And, that's actually all been driven by the fact that I think by and large CDNS hate media, and they want to pretend that all media or assets are in fact JPEGS or JavaScript files and so and so forth.
Dror Gill: 38:48 Or webpages.
Adrian Roe: 38:49 Exactly.
Dror Gill: 38:50 Yeah, like small chunks of data that's what they know how to handle best.
Adrian Roe: 38:52 Exactly. And so, the people distributing the content like to treat them as static assets, and they all have their infrastructures built around the very, very efficient delivery of static assets, and that creates high high latency. So, you then get technologies like WebRTC which is emerging, which we use heavily in production for ... So, one of our customers is a sports broadcaster, their customers can deliver their own live commentary on a system over WebRTC, and it basically doesn't add any latency to the process because while we'll hand off a low latency encoder of the feed over WebRTC to wherever the commentator is, the commentator will view the stream and commentate.
Adrian Roe: 39:34 In the meantime, we're going to a really high quality encode. In fact, this might be a mutual customer, but I probably won't say their name on air. We're going to do a really high quality encoder that same content in the meantime, and by the time we get the audio back from the commentator, we just mix that in with the crowd noise, add it to the video that we've already encoded at that point and away you go. And, you're pretty much getting live commentary on a system for free in terms of end to end latency. Yeah, and then sports, so we should be using WebRTC, we should be in this ...
Adrian Roe: 40:05 The problem, CDNS don't like WebRTC not at least because it's a connection oriented protocol. You can't just do the same for everybody. You've got to have separate encryption keys and it's all peer to peer and so on and so forth. And so, it doesn't scale using their standard models. And so, most of the discussion around low latency as far as I can tell is the extent to which you can pretend that your segmented assets are in fact live streams, and so Akamai has this thing where they'll start playing a segment before it's finished and so on and so forth. Well actually, it starts to look an awful lot like a progressive download at that point.
Mark Donnigan: 40:41 That's a great point. That's absolutely. Absolutely. And, what I find as I've walked around, like I said, walking around Streaming Media West, and looking at websites, reading marketing material, of everybody who has a low latency solution with a few exceptions, nobody's addressing the end to end factor of it. So, it cracks me up when I see an encoding vendor really touting low latency, low latency and I'm sitting here thinking, I mean Dror, what are we like 20 milliseconds? How much more low latency can you get than that?
Dror Gill: 41:19 Yeah, at the Kodak level it is very low.
Mark Donnigan: 41:21 Yeah, at the Kodak level. And then, when you begin to abstract out and of course the process adds time, right? But still, I mean the point is, is like it's ... I don't know, I guess part of what am reacting to and what I'm looking for, even in your response is that end to end, yes, but addressing latency end to end is really complicated because now just as you said, Adrian, now you have to look at the CDN, and you have to look at what you're doing on packaging, and you have to look at even your player architecture like progressive download. Some players can deal with that, great, other players can't. So, what do you do?
Dom Robinson: 42:04 So, one of the things that I think just stepping back and having a reasonably long game view of the evolution of the industry over here in, in the UK, particularly in Europe general, low latency has been a thing for 15, 20 years. And, the big thing that's changed and why low latencies all over the global US driven press is the deregulation of the gambling market, and that's why everyone's interested in low latency. Over here in the UK, we've had gambling online for live sports for 15, 20 years. And, for everyone ... I used to run a CDN from 2001 to end of the 2000s, and all the clients were interested in was fast start for advertising for VOD assets and low latency for betting delivery. And obviously, low latency is important because the lower the latency, the later you can shut your betting gates. And, if you've got a ten second segment or 30 seconds to an hour, three segments to wait, you've got to shut your betting maybe a minute, half a minute before the race finishes or before the race starts, whichever way you're doing the betting.
Dom Robinson: 43:14 And, that was very important over here. You didn't have a gambling market in the states online until last year I believe. And so, low latency just really wasn't very interesting. People were really only interested in can actually deliver reliably a big audience rather than can I deliver this to even small audiences, but with low latency, because I've got a betting market going on. And, as that betting deregulations come in, suddenly all the US centric companies have become really fascinated in whether they can shorten that low latency and so on and so forth. And, that's why companies 15, 20 years ago, over here, some of the big sports broadcast and so on, they were using RTMP extensively so that they could run their betting gates until the last second, and it really ramps up the amount of betting in those few seconds before the race starts.
Dom Robinson: 44:03 So, that's why it's important. It's not for any other reason. In fact, I sometimes rather sourly ask audiences if they really ever heard their neighbors cheering to a football game before they've seen it because being caught on a sweeney of socially gathering around the TV, and it's an important game like that where your neighbors might have have their TV on loud enough, you frankly got a TV and it's on as well.
Dom Robinson: 44:28 The real benchmark of the whole thing is can you beat the tweet, that's the measurable thing, and there's absurd little data in a tweet and a lot of tweets are machine generated, a goal is scored and it doesn't even take a fan in the stadium to type it, and send it to his friends, it's just instantly updated trying to beat a few packets of data across the world compared to trying to compress video, get it buffered, get it distributed across probably two or three stages of workflow decoded in the player and rendered. You're never going to be to tweet at that level. So, really the excitement is about betting, the deregulation of the betting market and gambling market.
Dror Gill: 45:06 So, that's interesting. Today you don't measure the latency between and over the air broadcast and the top over the internet broadcasts, but you want to beat another over the internet broadcast, which is a very small packets of the tweet. So.
Adrian Roe: 45:22 Exactly right.
Dror Gill: 45:23 Actually, competing with the social networks and other broadcast networks.
Dom Robinson: 45:26 Exactly.
Adrian Roe: 45:28 I can remember, there were tongue in cheek when WhatsApp were bought, they were boasting about the number of messages that they dealt with a day, and yeah it was very large number, billions of messages a day. And, I remembered a little back of an envelope calculation that if you ... Based on the adage that a picture was worth a thousand words, and across all the various different events and channels and live sports and stuff like that we cover, if you counted a thousand words for every frame of video that we delivered, we were two orders of magnitude higher than WhatsApp.
Dror Gill: 46:07 So, yeah. So, you had more traffic in your small company, you had more traffic than WhatsApp.
Adrian Roe: 46:11 Yeah.
Dror Gill: 46:13 A picture is worth a thousand words, and then you have 25 or 50 pictures every second. And, this is across all of your channels. So, yeah [crosstalk 00:46:22].
Mark Donnigan: 46:21 That's a lot of words. It maybe chuckle up. Well, this is-
Dror Gill: 46:27 We always say video is complicated and now we know why.
Mark Donnigan: 46:32 Exactly. Well, this has been an amazing discussion, and I think we should bring into a close with, I'd really like your perspective, Adrian and Dom, you're working with broadcasters and presumably sitting right in the middle of this OTT transition. Dom, I know you mentioned that for 20 years you'd been evangelizing IP, and now finally it's a thing, everybody gets it. But, just curious, maybe you can share with the listeners some trends that you're seeing, how is a traditional broadcast or someone who's operating a little more of your traditional infrastructure, et cetera, how are they adopting OTT into their workflows? Are they building parallel workflows? Are some fork lifting and making the full IP transition. I think this is a great conversation to end with.
Adrian Roe: 47:25 I think we're right at the cusp of exactly that. So, none of our customers are doing it side by side if they are full blown traditional broadcasters. I think increasingly a lot of our customers who may be deliver exclusively over the internet would also consider themselves broadcasters, and so I think the parlance is perhaps slightly out of date, but that's one of the things that I think is really interesting is some of the cultural challenges that come out of this. So, one of our customers who is a full blown traditional broadcaster, when you're dealing with fault tolerant large scale systems of the sort, that idea is built, then one of the things that's a given is that it's going to be a computer that decides which server is going to be responsible for which particular, this is BBC one's encoder, this is ... Yeah, whatever ITVs encoder or whatever. It's going to be a computer that makes those decisions because a computer can react in milliseconds if one of those services is no longer available and reroute it somewhere else.
Adrian Roe: 48:28 And, this wasn't a public cloud implementation it was a private cloud implementation that they had couple of racks of servers and data management infrastructure on top that was doing all of the dynamic allocation and tolerance and all this clever stuff. And they said, so when we're showing our customers around, if channel four comes around, how can we tell then which is their encoder? And we said, you count. There isn't a channel four encoder there's an encoder that might be doing the job.
Adrian Roe: 48:55 And, one of the features we had to add to the product as just to get over the cultural hurdle with them was the concept of a preferred encoder. So, if everything was in its normal happy state, then yeah, this particular encoder, halfway down on the right hand side of rack three, was going to be the one doing channel four, and just those simple things where they think people do still think in terms of appliances and raw rian and so on and so forth, and some of the challenges to move away from that into cloud thinking bit actually on the cloud or not, cloud thinking still applies it. It's funny where people trip up.
Dom Robinson: 49:36 One of my bugbears in the industry, I'm a bit of a pedant with some of the terminology that gets used and so on. One of my bugbears is the term OTT. So, having spent a good long while playing with video and audio distribution over IP networks and so on, I struggle to think of any broadcast technology, which doesn't use IP at some point in this either production or distribution workflow, there just isn't any now. And so, if you're watching live news, the contribution visa coming over cell phones which are contribution is some sort of streaming protocol or a film or TV program production people are emailing files or they're dropboxing files, or they're sending them through digital asset management systems or however it may be.
Dom Robinson: 50:20 But, the programs are being created using IP and have been for quite a while and increasingly nobody replaces technology with some sort of proprietary non IP based tool these days at any level in the broadcast industry. I rather store everything I can to try to avoid using the word OTT. And being a pedant about it, OTT simply means the paywall is outside of this last mile access network. That's all it means. It has nothing whatsoever to do with video distribution or streaming or anything like that. It's simply to do with where you take your payment from somebody.
Dom Robinson: 50:57 So, Netflix has a hybridized side, but Netflix, you generally access through an ISP and when you make your payment, you pay Netflix directly. You don't pay through your ISP, that is an OTT service. Skype is an OTT service. Again, you connect through your phone service, your cable service, whatever it may be, but you actually subscribe directly with Skype, that is a true OTT service, and that's what OTT means. It's become in the last eight years synonymous with streaming ,and I can't think of a broadcast network which doesn't at some point use IP either streaming or file transfer based technologies to compose the program.
Dom Robinson: 51:37 So, broadcast is streaming, streaming is broadcast. They have been synonymous for over a decade. It is how you connect the payment, which defines something as OTT, and it may well be that you can receive a video stream outside of one particular ISPs network, but that doesn't really mean anything. So, this battle between broadcast and OTT, it's a meaningless decision of where you're collecting payments for me. It really doesn't have any bearing on the technologies that we all work with which are video compression and distribution and so on. So.
Mark Donnigan: 52:11 That's brilliant. That is really, really a smart observation and analysis there Dom. Well, I think we should wrap it up here. We definitely need to do a part two. I think we will have you guys back, there's so much more we could be talking about, but I want to thank our amazing audience, without you the Video Insiders Podcast would just be Dror and me talking to ourselves.
Dror Gill: 52:38 Buzzing to ourselves some buzzy words.
Mark Donnigan: 52:40 Buzzy words, buzzing, buzzing, taking up bits on a server somewhere and this has been a production of Beamer Imaging Limited, you can subscribe at thevideoinsiders.com where you can listen to us on Spotify, on iTunes, on Google Play, and more platforms coming soon. And, if you'd like to try out Beamer Codecs in your lab or production environment, we're actually giving away up to 100 hours of HEVC and H.264 encoding every month. Just go to beamer.com/free, that's F-R-E-E to get started. And until next time, thank you and have an awesome day encoding video.
Speaker 1: 53:30 Thank you for listening to the Video Insiders Podcast, a production of Beamr Limited. To begin using Beamrs' Codecs today, go to https://beamr.com/free to receive up to 100 hours of no cost HEVC and H.264 trans coding every month.
