codecs

Yurong Jiang LinkedIn profile

LinkedIn Engineering blog post about video conferencing 

---------------------------------------------------

Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.

Email thevideoinsiders@beamr.com to be a guest on the show.

Learn more about Beamr

Jan Ozer LinkedIn profile

Streaming Learning Center website

Jan Ozer on Streaming Media Magazine

Jan Ozer on OTTVerse

---------------------------------------------------

Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.

Email thevideoinsiders@beamr.com to be a guest on the show.

Learn more about Beamr

Zoe Liu, who is the founder and President of Visionular, dives into the details of emerging video codecs and formats with Prajanma Singh, Senior Product Marketing Manager for Cloudinary. They discuss codec standards, such as H.266/Versatile Video Coding (VCC), Essential Video Coding (EVC), and Low Complexity Enhancement Video Coding (LCEVC), and many details dealing with overall video compression.

Learn More about Cloudinary and Visionular

• Cloudinary
• Visionular

Learn More about Video Delivery, Codec and Standards

• Cloudinary Documentation on Video Transformations
• AV1 Standard Information from Visionular
• Versatile Video Coding (VVC) from the Fraunhofer Heinrich Hertz Institute
• Presentation on EVC from Jonatan Samuelsson

Follow Prajanma and Zoe

• Prajanma on LinkedIn
• Zoe Liu on LinkedIn

Show music provided by RIOT: https://www.youtube.com/playlist?list=PLKCo4y0JbV2ZW3W9s3u0WoGjAnFKZlzi1

After you're done listening, share this episode on your favorite social media channels and then send an email to support@cloudinary.com with a link to the post you shared. By doing so, you can earn one additional credit for your Cloudinary plan!

Not a Cloudinary user yet? Sign up for free!

Click to watch SPIE Future Video Codec Panel Discussion

Related episode with Gary Sullivan at Microsoft: VVC, HEVC & other MPEG codec standards

Interview with MPEG Chairman Leonardo Charliogne: MPEG Through the Eyes of it's Chairman

Learn about FastDVO here

Pankaj Topiwala LinkedIn profile

--------------------------------------

The Video Insiders LinkedIn Group is where thousands of your peers are discussing the latest video technology news and sharing best practices. Click here to join

Would you like to be a guest on the show? Email: thevideoinsiders@beamr.com

Learn more about Beamr

--------------------------------------

TRANSCRIPT:

Pankaj Topiwala: 00:00 With H.264 H.265 HEVC in 2013, we were now able to do up to 300 to one to up to 500 to one compression on a, let's say a 4K video. And with VVC we have truly entered a new realm where we can do up to 1000 to one compression, which is three full orders of magnitude reduction of the original size. If the original size is say 10 gigabits, we can bring that down to 10 megabits. And that's unbelievable. And so video compression truly is a remarkable technology and you know, it's a, it's a marval to look at

Announcer: 00:39 The Video Insiders is the show that makes sense of all that is happening in the world of online video as seen through the eyes of a second generation codec nerd and a marketing guy who knows what I-frames and macro blocks are. And here are your hosts, Mark Donnigan and Dror Gill.

Speaker 3: 00:39

Dror Gill: 01:11 Today we're going to talk with one of the key figures in the development of a video codecs and a true video insider Pankaj Topiwala. Hello Pankaj and welcome to The Video Insiders podcast.

Pankaj Topiwala: 01:24 Gentlemen. hello, and thank you very much for this invite. It looks like it's going to be a lot of fun.

Mark Donnigan: 01:31 It is. Thank you for joining Pankaj.

Dror Gill: 01:33 Yeah, it sure will be a lot of fun. So can you start by telling us a little bit about your experience in codec development?

Pankaj Topiwala: 01:41 Sure, so, I should say that unlike a number of the other people that you have interviewed or may interview my background is fair bit different. I really came into this field really by a back door and almost by chance my degree PhD degree is actually in mathematical physics from 1985. And I actually have no engineering, computer science or even management experience. So naturally I run a small research company working in video compression and analytics, and that makes sense, but that's just the way things go in the modern world. But that the effect for me was a, and the entry point was that even though I was working in very, very abstract mathematics I decided to leave. I worked in academia for a few years and then I decided to join industry. And at that point they were putting me into applied mathematical research.

Pankaj Topiwala: 02:44 And the topic at that time that was really hot in applied mathematics was a topic of wavelets. And I ended up writing and edited a book called wavelet image and video compression in 1998. Which was a lot of fun along with quite a few other co authors on that book. But, wavelets had its biggest contribution in the compression of image and video. And so that led me finally to enter into, and I noticed that video compression was a far larger field than image compression. I mean, by many orders, by orders of magnitude. It is probably a hundred times bigger in terms of market size than, than image compression. And as a result I said, okay, if the sexiest application of this new fangled mathematics could be in video compression I entered that field roughly with the the book that I mentioned in 1998.

Mark Donnigan: 03:47 So one thing that I noticed Pankaj cause it's really interesting is your, your initial writing and you know, research was around wavelet compression and yet you have been very active in ISO MPEG, all block-based codecs. So, so tell us about that?

Pankaj Topiwala: 04:08 Okay. Well obviously you know when you make the transition from working on the wavelets and our initial starting point was in doing wavelet based video compression. When I started first founded my company fastVDO in 1998, 1999 period we were working on wavelet based video compression and we, we pushed that about as much as we could. And at that, at one point we had what we felt was the world's best a video compression using wavelets in fact, but best overall. And it had the feature that you know, one thing that we should, we should tell your view or reader listeners is that the, the value of wavelets in particular in image coding is that not only can you do state of the art image coding, but you can make the bitstream what is called embedded, meaning you can chop it off at anywhere you like, and it's still a decodable stream.

Pankaj Topiwala: 05:11 And in fact it is the best quality you can get for that bit rate. And that is a powerful, powerful thing you can do in image coding. Now in video, there is actually no way to do that. Video is just so much more complicated, but we did the best we could to make it not embedded, but at least scalable. And we, we built a scalable wavelet based video codec, which at that time was beating at the current implementations of MPEG4. So we were very excited that we could launch a company based on a proprietary codec that was based on this new fangled mathematics called wavelets. And lead us to a state of the art codec. The facts of the ground though is that just within the first couple of years of running our company, we found that in fact the block-based transformed codecs that everybody else was using, including the implementers of MPEG4.

Pankaj Topiwala: 06:17 And then later AVC, those quickly surpassed anything we could build with with wavelets in terms of both quality and stability. The wavelet based codecs were not as powerful or as stable. And I can say quite a bit more about why that's true. If you want?

Dror Gill: 06:38 So when you talk about stability, what exactly are you referring to in, in a video codec?

Pankaj Topiwala: 06:42 Right. So let's let's take our listeners back a bit to compare image coding and video coding. Image coding is basically, you're given a set of pixels in a rectangular array and we normally divide that into blocks of sub blocks of that image. And then do transforms and then quantization and than entropy coding, that's how we typically do image coding. With the wavelet transform, we have a global transform. It's a, it's ideally done on the entire image.

Pankaj Topiwala: 07:17 And then you could do it multiple times, what are called multiple scales of the wavelet transform. So you could take various sub sub blocks that you create by doing the wavelet transfer and the low pass high pass. Ancs do that again to the low low pass for multiple scales, typically about four or five scales that are used in popular image codecs that use wavelets. But now in video, the novelty is that you don't have one frame. You have many, many frames, hundreds or thousands or more. And you have motion. Now, motion is something where you have pieces of the image that float around from one frame to another and they float randomly. That is, it's not as if all of the motion is in one direction. Some things move one way, some things move other ways, some things actually change orientations.

Pankaj Topiwala: 08:12 And they really move, of course, in three dimensional space, not in our two dimensional space that we capture. That complicates video compression enormously over image compression. And it particularly complicates all the wavelet methods to do video compression. So, wavelet methods that try to deal with motion were not very successful. The best we tried to do was using motion compensated video you know, transformed. So doing wavelet transforms in the time domain as well as the spatial domain along the paths of motion vectors. But that was not very successful. And what I mean by stability is that as soon as you increase the motion, the codec breaks, whereas in video coding using block-based transforms and block-based motion estimation and compensation it doesn't break. It just degrades much more gracefully. Wavelet based codecs do not degrade gracefully in that regard.

Pankaj Topiwala: 09:16 And so we of course, as a company we decided, well, if those are the facts on the ground. We're going to go with whichever way video coding is going and drop our initial entry point, namely wavelets, and go with the DCT. Now one important thing we found was that even in the DCT- ideas we learned in wavelets can be applied right to the DCT. And I don't know if you're familiar with this part of the story, but a wavelet transform can be decomposed using bits shifts and ads only using something called the lifting transform, at least a important wavelet transforms can. Now, it turns out that the DCT can also be decomposed using lifting transforms using only bit shifts and ads. And that is something that my company developed way back back in 1998 actually.

Pankaj Topiwala: 10:18 And we showed that not only for DCT, but a large class of transforms called lab transforms, which included the block transforms, but in particular included more powerful transforms the importance of that in the story of video coding. Is that up until H.264, all the video codec. So H.261, MPEG-1, MPEG-2, all these video codecs used a floating point implementation of the discrete cosign transform and without requiring anybody to implement you know a full floating point transform to a very large number of decimal places. What they required then was a minimum accuracy to the DCT and that became something that all codecs had to do. Instead. If you had an implementation of the DCT, it had to be accurate to the true floating point DCT up to a certain decimal point in, in the transform accuracy.

Pankaj Topiwala: 11:27 With the advent of H.264, with H.264, we decided right away that we were not going to do a flooding point transform. We were going to do an integer transform. That decision was made even before I joined, my company joined, the development base, H.264, AVC, But they were using 32 point transforms. We found that we could introduce 16 point transforms, half the complexity. And half the complexity only in the linear dimension when you, when you think of it as a spatial dimension. So two spatial dimensions, it's a, it's actually grows more. And so the reduction in complexity is not a factor of two, but at least a factor of four and much more than that. In fact, it's a little closer to exponential. The reality is that we were able to bring the H.264 codec.

Pankaj Topiwala: 12:20 So in fact, the transform was the most complicated part of the entire codec. So if you had a 32 point transform, the entire codec was at 32 point technology and it needed 32 points, 32 bits at every sample to process in hardware or software. By changing the transform to 16 bits, we were able to bring the entire codec to a 16 bit implementation, which dramatically improved the hardware implementability of this transfer of this entire codec without at all effecting the quality. So that was an important development that happened with AVC. And since then, we've been working with only integer transforms.

Mark Donnigan: 13:03 This technical history is a really amazing to hear. I, I didn't actually know that Dror or you, you probably knew that, but I didn't.

Dror Gill: 13:13 Yeah, I mean, I knew about the transform and shifting from fixed point, from a floating point to integer transform. But you know, I didn't know that's an incredible contribution Pankaj.

Pankaj Topiwala: 13:27 We like to say that we've saved the world billions of dollars in hardware implementations. And we've taken a small a small you know, a donation as a result of that to survive as a small company.

Dror Gill: 13:40 Yeah, that's great. And then from AVC you moved on and you continued your involvement in, in the other standards, right? That's followed.

Pankaj Topiwala: 13:47 in fact, we've been involved in standardization efforts now for almost 20 years. My first meeting was a, I recall in may of 2000, I went to a an MPEG meeting in Geneva. And then shortly after that in July I went to an ITU VCEG meeting. VCEG is the video coding experts group of the ITU. And MPEG is the moving picture experts group of ISO. These two organizations were separately pursuing their own codecs at that time.

Pankaj Topiwala: 14:21 ISO MPEG was working on MPEG-4 and ITU VCEG was working on H.263, and 263 plus and 263 plus plus. And then finally they started a project called 263 L for longterm. And eventually it became clear to these two organizations that look, it's silly to work on, on separate codecs. They had worked once before in MPEG-2 develop a joint standard and they decided to, to form a joint team at that time called the joint video team, JVT to develop the H.264 AVC video codec, which was finally done in 2003. We participate participated you know fully in that making many contributions of course in the transform but also in motion estimation and other aspects. So, for example, it might not be known that we also contributed the fast motion estimation that's now widely used in probably nearly all implementations of 264, but in 265 HEVC as well.

Pankaj Topiwala: 15:38 And we participated in VVC. But one of the important things that we can discuss is these technologies, although they all have the same overall structure, they have become much more complicated in terms of the processing that they do. And we can discuss that to some extent if you want?

Dror Gill: 15:59 The compression factors, just keep increasing from generation to generation and you know, we're wondering what's the limit of that?

Pankaj Topiwala: 16:07 That's of course a very good question and let me try to answer some of that. And in fact that discussion I don't think came up in the discussion you had with Gary Sullivan, which certainly could have but I don't recall it in that conversation. So let me try to give for your listeners who did not catch that or are not familiar with it. A little bit of the story.

Pankaj Topiwala: 16:28 The first international standard was the ITU. H.261 standard dating roughly to 1988 and it was designed to do only about 15 to one to 20 to one compression. And it was used mainly for video conferencing. And at that time you'd be surprised from our point of view today, the size of the video being used was actually incredibly tiny about QCIP or 176 by 144 pixels. Video of that quality that was the best we could conceive. And we thought we were doing great. And doing 20 to one compression, wow! Recall by the way, that if you try to do a lossless compression of any natural signal, whether it's speech or audio or images or video you can't do better than about two to one or at most about two and a half to one.

Pankaj Topiwala: 17:25 You cannot do, typically you cannot even do three to one and you definitely cannot do 10 to one. So a video codec that could do 20 to one compression was 10 times better than what you could do lossless, I'm sorry. So this is definitely lossy, but lossy with still a good quality so that you can use it. And so we thought we were really good. When MPEG-1 came along in, in roughly 1992 we were aiming for 25 to one compression and the application was the video compact disc, the VCD. With H.262 or MPEG-2 roughly 1994, we were looking to do about 35 to one compression, 30 to 35. And the main application was then DVD or also broadcast television. At that point, broadcast television was ready to use at least in some, some segments.

Pankaj Topiwala: 18:21 Try digital broadcasting. In the United States, that took a while. But in any case it could be used for broadcast television. And then from that point H.264 AVC In 2003, we jumped right away to more than 100 to one compression. This technology at least on large format video can be used to shrink the original size of a video by more than two orders of magnitude, which was absolutely stunning. You know no other natural signal, not speech, not broadband, audio, not images could be compressed that much and still give you high quality subjective quality. But video can because it's it is so redundant. And because we don't understand fully yet how to appreciate video. Subjectively. We've been trying things you know, ad hoc. And so the entire development of video coding has been really by ad hoc methods to see what quality we can get.

Pankaj Topiwala: 19:27 And by quality we been using two two metrics. One is simply a mean square error based metric called peak signal to noise ratio or PSNR. And that has been the industry standard for the last 35 years. But the other method is simply to have people look at the video, what we call subjective rating of the video. Now it's hard to get a subjective rating. That's reliable. You have to do a lot of standardization get a lot of different people and take mean opinion scores and things like that. That's expensive. Whereas PSNR is something you can calculate on a computer. And so people have mostly in the development of video coding for 35 years relied on one objective quality metric called PSNR. And it is good but not great. And it's been known right from the beginning that it was not perfect, not perfectly correlated to video quality, and yet we didn't have anything better anyway.

Pankaj Topiwala: 20:32 To finish the story of the video codecs with H.265 HEVC in 2013, we were now able to do up to 300 to one to up to 500 to one compression on let's say a 4K. And with VVC we have truly entered a new realm where we can do up to 1000 to one compression, which is three full orders of magnitude reduction of the original size. If the original size is say, 10 gigabits, we can bring that down to 10 megabits. And that's unbelievable. And so video compression truly is a remarkable technology. And you know, it's a, it's a marvel to look at. Of course it does not, it's not magic. It comes with an awful lot of processing and an awful lot of smarts have gone into it. That's right.

Mark Donnigan: 21:24 You know Pankaj, that, is an amazing overview and to hear that that VVC is going to be a thousand to one. You know, compression benefit. Wow. That's incredible!

Pankaj Topiwala: 21:37 I think we should of course we should of course temper that with you know, what people will use in applications. Correct. They may not use the full power of a VVC and may not crank it to that level. Sure, sure. I can certainly tell you that that we and many other companies have created bitstreams with 1000 to one or more compression and seeing video quality that we thought was usable.

Mark Donnigan: 22:07 One of the topics that has come to light recently and been talked about quite a bit. And it was initially raised by Dave Ronca who used to lead encoding at Netflix for like 10 years. In fact you know, I think he really built that department, the encoding team there and is now at Facebook. And he wrote a LinkedIn article post that was really fascinating. And what he was pointing out in this post was, was that with compression efficiency and as each generation of codec is getting more efficient as you just explained and gave us an overview. There's a, there's a problem that's coming with that in that each generation of codec is also getting even more complex and you know, in some settings and, and I suppose you know, Netflix is maybe an example where you know, it's probably not accurate to say they have unlimited compute, but their application is obviously very different in terms of how they can operate their, their encoding function compared to someone who's doing live, live streaming for example, or live broadcast. Maybe you can share with us as well. You know, through the generation generational growth of these codecs, how has the, how has the compute requirements also grown and has it grown in sort of a linear way along with the compression efficiency? Or are you seeing, you know, some issues with you know, yes, we can get a thousand to one, but our compute efficiency is getting to the, where we could be hitting a wall.

Pankaj Topiwala: 23:46 You asked a good question. Has the complexity only scaled linearly with the compression ratio? And the answer is no. Not at all. Complexity has outpaced the compression ratio. Even though the compression ratio is, is a tremendous, the complexity is much, much higher. And has always been at every step. First of all there's a big difference in doing the research, the research phase in development of the, of a technology like VVC where we were using a standardized reference model that the committee develops along the way, which is not at all optimized. But that's what we all use because we share a common code base. And make any new proposals based on modifying that code base. Now that code base is always along the entire development chain has always been very, very slow.

Pankaj Topiwala: 24:42 And true implementations are anywhere from 100 to 500 times more efficient in complexity than the reference software. So right away you can have the reference software for say VVC and somebody developing a, an implementation that's a real product. It can be at least 100 times more efficient than what the reference software, maybe even more. So there's a big difference. You know, when we're developing a technology, it is very hard to predict what implementers will actually come up with later. Of course, the only way they can do that is that companies actually invest the time and energy right away as they're developing the standard to build prototype both software and hardware and have a good idea that when they finish this, you know, what is it going to really cost? So just to give you a, an idea, between, H.264 and

Pankaj Topiwala: 25:38 H.265, H.264, only had two transforms of size, four by four and eight by eight. And these were integer transforms, which are only bit shifts and adds, took no multiplies and no divides. The division in fact got incorporated into the quantizer and as a result, it was very, very fast. Moreover, if you had to do, make decisions such as inter versus intra mode, the intra modes there were only about eight or 10 intra modes in H.264. By contrast in H.265. We have not two transforms eight, four by four and eight by, but in fact sizes of four, eight, 16 and 32. So we have much larger sized transforms and instead of a eight or 10 intra modes, we jumped up to 35 intra modes.

Pankaj Topiwala: 26:36 And then with a VVC we jumped up to 67 intro modes and we just, it just became so much more complex. The compression ratio between HEVC and VVC is not quite two to one, but let's say, you know, 40% better. But the the complexity is not 40% more. On the ground and nobody has yet, to my knowledge, built a a, a, a fully compliant and powerful either software or hardware video codec for VVC yet because it's not even finished yet. It's going to be finished in July 2020. When it, when, the dust finally settles maybe four or five years from now, it will be, it will prove to be at least three or four times more complex than HEVC encoder the decoder, not that much. The decoder, luckily we're able to build decoders that are much more linear than the encoder.

Pankaj Topiwala: 27:37 So I guess I should qualify as discussion saying the complexity growth is all mostly been in the encoder. The decoder has been a much more reasonable. Remember, we are always relying on this principle of ever-increasing compute capability. You know, a factor of two every 18 months. We've long heard about all of this, you know, and it is true, Moore's law. If we did not have that, none of this could have happened. None of this high complexity codecs, whatever had been developed because nobody would ever be able to implement them. But because of Moore's law we can confidently say that even if we put out this very highly complex VVC standard, someday and in the not too distant future, people will be able to implement this in hardware. Now you also asked a very good question earlier, is there a limit to how much we can compress?

Pankaj Topiwala: 28:34 And also one can ask relatively in this issue, is there a limit to a Moore's law? And we've heard a lot about that. That may be finally after decades of the success of Moore's law and actually being realized, maybe we are now finally coming to quantum mechanical limits to you know how much we can miniaturize in electronics before we actually have to go to quantum computing, which is a totally different you know approach to doing computing because trying to go smaller die size. Well, we'll make it a unstable quantum mechanically. Now the, it appears that we may be hitting a wall eventually we haven't hit it yet, but we may be close to a, a physical limit in die size. And in the observations that I've been making at least it seems possible to me that we are also reaching a limit to how much we can compress video even without a complexity limit, how much we can compress video and still obtain reasonable or rather high quality.

Pankaj Topiwala: 29:46 But we don't know the answer to that. And in fact there are many many aspects of this that we simply don't know. For example, the only real arbiter of video quality is subjective testing. Nobody has come up with an objective video quality metric that we can rely on. PSNR is not it. When, when push comes to shove, nobody in this industry actually relies on PSNR. They actually do subjective testing well. So in that scenario, we don't know what the limits of visual quality because we don't understand human vision, you know, we try, but human vision is so complicated. Nobody can understand the impact of that on video quality to any very significant extent. Now in fact, the first baby steps to try to understand, not explicitly but implicitly capture subjective human video quality assessment into a neural model. Those steps are just now being taken in the last couple of years. In fact, we've been involved, my company has been involved in, in getting into that because I think that's a very exciting area.

Dror Gill: 30:57 I tend to agree that modeling human perception with a neural network seems more natural than, you know, just regular formulas and algorithms which are which are linear. Now I, I wanted to ask you about this process of, of creating the codecs. It's, it's very important to have standards. So you encode a video once and then you can play it anywhere and anytime and on any device. And for this, the encoder and decoder need to agree on exactly the format of the video. And traditionally you know, as you pointed out with all the history of, of development. Video codecs have been developed by standardization bodies, MPEG and ITU first separately. And then they joined forces to develop the newest video standards. But recently we're seeing another approach to develop codecs, which is by open sourcing them.

Dror Gill: 31:58 Google started with an open source code, they called VP9 which they first developed internally. Then they open sourced it and and they use it widely across their services, especially in, YouTube. And then they joined forces with the, I think the largest companies in the world, not just in video but in general. You know those large internet giants such as Amazon and Facebook and and Netflix and even Microsoft, Apple, Intel have joined together with the Alliance of Open Media to jointly create another open codec called AV1. And this is a completely parallel process to the MPEG codec development process. And the question is, do you think that this was kind of a one time effort to, to to try and find a, or develop a royalty free codec, or is this something that will continue? And how do you think the adoption of the open source codecs versus the committee defined codecs, how would that adoption play out in the market?

Pankaj Topiwala: 33:17 That's of course a large topic on its own. And I should mention that there have been a number of discussions about that topic. In particular at the SPIE conference last summer in San Diego, we had a panel discussion of experts in video compression to discuss exactly that. And one of the things we should provide to your listeners is a link to that captured video of the panel discussion where that topic is discussed to some significant extent. And it's on YouTube so we can provide a link to that. My answer. And of course none of us knows the future. Right. But we're going to take our best guesses. I believe that this trend will continue and is a new factor in the landscape of video compression development.

Pankaj Topiwala: 34:10 But we should also point out that the domain of preponderance use preponderant use of these codecs is going to be different than in our traditional codecs. Our traditional codecs such as H.264 265, were initially developed for primarily for the broadcast market or for DVD and Blu-ray. Whereas these new codecs from AOM are primarily being developed for the streaming media industry. So the likes of Netflix and Amazon and for YouTube where they put up billions of user generated videos. So, for the streaming application, the decoder is almost always a software decoder. That means they can update that decoder anytime they do a software update. So they're not limited by a hardware development cycle. Of course, hardware companies are also building AV1.

Pankaj Topiwala: 35:13 And the point of that would be to try to put it into handheld devices like laptops, tablets, and especially smartphones. But to try to get AV1 not only as a decoder but also as an encoder in a smartphone is going to be quite complicated. And the first few codecs that come out in hardware will be of much lower quality, for example, comparable to AVC and not even the quality of HEVC when they first start out. So that's... the hardware implementations of AV1 that work in real time are not going to be, it's going to take a while for them to catch up to the quality that AV1 can offer. But for streaming we, we can decode these streams reasonably well in software or in firmware. And the net result is that, or in GPU for example, and the net result is that these companies can already start streaming.

Pankaj Topiwala: 36:14 So in fact Google is already streaming some test streams maybe one now. And it's cloud-based YouTube application and companies like Cisco are testing it already, even for for their WebEx video communication platform. Although the quality will not be then anything like the full capability of AV1, it'll be at a much reduced level, but it'll be this open source and notionally, you know, royalty free video codec.

Dror Gill: 36:50 Notionally. Yeah. Because they always tried to do this, this dance and every algorithm that they try to put into the standard is being scrutinized and, and, and they check if there are any patents around it so they can try and keep this notion of of royalty-free around the codec because definitely the codec is open source and royalty free.

Dror Gill: 37:14 I think that is, is, is a big question. So much IP has gone into the development of the different MPEG standards and we know it has caused issues. Went pretty smoothly with AVC, with MPEG-LA that had kind of a single point of contact for licensing all the essential patents and with HEVC, that hasn't gone very well in the beginning. But still there is a lot of IP there. So the question is, is it even possible to have a truly royalty free codec that can be competitive in, in compression efficiency and performance with the codec developed by the standards committee?

Pankaj Topiwala: 37:50 I'll give you a two part answer. One because of the landscape of patents in the field of video compression which I would describe as being, you know very, very spaghetti like and patents date back to other patents.

Pankaj Topiwala: 38:09 And they cover most of the, the topics and the most of the, the tools used in video compression. And by the way we've looked at the AV1 and AV1 is not that different from all the other standards that we have. H.265 or VVC. There are some things that are different. By and large, it resembles the existing standards. So can it be that this animal is totally patent free? No, it cannot be that it is patent free. But patent free is not the same as royalty free. There's no question that AV1 has many, many patents, probably hundreds of patents that reach into it. The question is whether the people developing and practicing AV1 own all of those patents. That is of course, a much larger question.

Pankaj Topiwala: 39:07 And in fact, there has been a recent challenge to that, a group has even stood up to proclaim that they have a central IP in AV1. The net reaction from the AOM has been to develop a legal defense fund so that they're not going to budge in terms of their royalty free model. If they do. It would kill the whole project because their main thesis is that this is a world do free thing, use it and go ahead. Now, the legal defense fund then protects the members of that Alliance, jointly. Now, it's not as if the Alliance is going to indemnify you against any possible attack on IP. They can't do that because nobody can predict, you know, where somebody's IP is. The world is so large, so many patents in that we're talking not, not even hundreds and thousands, but tens of thousands of patents at least.

Pankaj Topiwala: 40:08 So nobody in the world has ever reviewed all of those patent. It's not possible. And the net result is that nobody can know for sure what technology might have been patented by third parties. But the point is that because such a large number of powerful companies that are also the main users of this technology, you know, people, companies like Google and Apple and Microsoft and, and Netflix and Amazon and Facebook and whatnot. These companies are so powerful. And Samsung by the way, has joined the Alliance. These companies are so powerful that you know, it would be hard to challenge them. And so in practice, the point is they can project a royalty-free technology because it would be hard for anybody to challenge it. And so that's the reality on the ground.

Pankaj Topiwala: 41:03 So at the moment it is succeeding as a royalty free project. I should also point out that if you want to use this, not join the Alliance, but just want to be a user. Even just to use it, you already have to offer any IP you have in this technology it to the Alliance. So all users around the world, so if tens of thousands and eventually millions of you know, users around the world, including tens of thousands of companies around the world start to use this technology, they will all have automatically yielded any IP they have in AV1, to the Alliance.

Dror Gill: 41:44 Wow. That's really fascinating. I mean, first the distinction you made between royalty free and patent free. So the AOM can keep this technology royalty free, even if it's not patent free because they don't charge royalties and they can help with the legal defense fund against patent claim and still keep it royalty free. And, and second is the fact that when you use this technology, you are giving up any IP claims against the creators of the technology, which means that if any, any party who wants to have any IP claims against the AV1 encoder cannot use it in any form or shape.

Pankaj Topiwala: 42:25 That's at least my understanding. And I've tried to look at of course I'm not a lawyer. And you have to take that as just the opinion of a video coding expert rather than a lawyer dissecting the legalities of this. But be that as it may, my understanding is that any user would have to yield any IP they have in the standard to the Alliance. And the net result will be if this technology truly does get widely used more IP than just from the Alliance members will have been folded into into it so that eventually it would be hard for anybody to challenge this.

Mark Donnigan: 43:09 Pankaj, what does this mean for the development of so much of the technology has been in has been enabled by the financial incentive of small groups of people, you know, or medium sized groups of people forming together. You know, building a company, usually. Hiring other experts and being able to derive some economic benefit from the research and the work and the, you know, the effort that's put in. If all of this sort of consolidates to a handful or a couple of handfuls of, you know, very, very large companies, you know, does that, I guess I'm, I'm asking from your view, will, will video and coding technology development and advancements proliferate? Will it sort of stay static? Because basically all these companies will hire or acquire, you know, all the experts and you know, it's just now everybody works for Google and Facebook and Netflix and you know... Or, or do you think it will ultimately decline? Because that's something that that comes to mind here is, you know, if the economic incentives sort of go away, well, you know, people aren't going to work for free!

Pankaj Topiwala: 44:29 So that's of course a, another question and a one relevant. In fact to many of us working in video compression right now, including my company. And I faced this directly back in the days of MPEG-2. There was a two and a half dollar ($2.50) per unit license fee for using MPEG-2. That created billions of dollars in licensing in fact, the patent pool, MPEG-LA itself made billions of dollars, even though they took only 10% of the proceeds, they already made billions of dollars, you know, huge amounts of money. With the advent of H.264 AVC, the patent license went not to from two and a half dollars to 25 cents a unit. And now with HEVC, it's a little bit less than that per unit. Of course the number of units has grown exponentially, but then the big companies don't continue to pay per unit anymore.

Pankaj Topiwala: 45:29 They just pay a yearly cap. For example, 5 million or 10 million, which to these big companies is is peanuts. So there's a yearly cap for the big companies that have, you know, hundreds of millions of units. You know imagine the number of Microsoft windows that are out there or the number of you know, Google Chrome browsers. And if you have a, a codec embedded in the browser there are hundreds of millions of them, if not billions of them. And so they just pay a cap and they're done with it. But even then, there was up till now an incentive for smart engineers to develop exciting new ideas in a future video coding. But, and that has been up the story up till now. But when, if it happens that this AOM model with AV1 and then AV2, really becomes a dominant codec and takes over the market, then there will be no incentive for researchers to devote any time and energy.

Pankaj Topiwala: 46:32 Certainly my company for example, can't afford to you know, just twiddle thumbs, create technologies for which there is absolutely no possibility of a royalty stream. So we, we cannot be in the business of developing video coding when video coding doesn't pay. So the only thing that makes money, is Applications, for example, a streaming application or some other such thing. And so Netflix and, and Google and Amazon will be streaming video and they'll charge you per stream but not on the codec. So that that's an interesting thing and it certainly affects the future development of video. It's clear to me it's a negative impact on the research that we got going in. I can't expect that Google and Amazon and Microsoft are going to continue to devote the same energy to develop future compression technologies in their royalty free environment that companies have in the open standards development technology environment.

Pankaj Topiwala: 47:34 It's hard for me to believe that they will devote that much energy. They'll devote energy, but it will not be the the same level. For example, in developing a video standards such as HEVC, it took up to 10 years of development by on the order of 500 to 600 experts, well, let's say four to 500 experts from around the world meeting four times a year for 10 years.

Mark Donnigan: 48:03 That is so critical. I want you to repeat that again.

Pankaj Topiwala: 48:07 Well, I mean so very clearly we've been putting out a video codec roughly on the schedule of once every 10 years. MPEG-2 was 1994. AVC was 2003 and also 2004. And then HEVC in 2013. Those were roughly 10 years apart. But VVC we've accelerated the schedule to put one out in seven years instead of 10 years. But even then you should realize that we had been working right since HEVC was done.

Pankaj Topiwala: 48:39 We've been working all this time to develop VVC and so on the order of 500 experts from around the world have met four times a year at all international locations, spending on the order of $100 million per meeting. You know so billions of dollars have been spent by industry to create these standards, many billions and it can't happen, you know without that. It's hard for me to believe that companies like Microsoft, Google, and whatnot, are going to devote billions to develop their next incremental, you know, AV1and AV2 AV3's. But maybe they will it just, that there's no royalty stream coming from the codec itself, only the application. Then the incentive, suppose they start dominating to create even better technology will not be there. So there really is a, a financial issue in this and that's at play right now.

Dror Gill: 49:36 Yeah, I, I find it really fascinating. And of course, Mark and I are not lawyers, but all this you know, royalty free versus committee developed open source versus a standard those large companies who some people fear, you know, their dominance and not only in video codec development, but in many other areas. You know, versus you know, dozens of companies and hundreds of engineers working for seven or 10 years in a codec. So you know, it's really different approaches different methods of development eventually to approach the exact same problem of video compression. And, and how this turns out. I mean we, we cannot forecast for sure, but it will be very interesting, especially next year in 2020 when VVC is ratified. And at around the same time, EVC is ratified another codec from the MPEG committee.

Dror Gill: 50:43 And then AV1, and once you know, AV1 starts hitting the market. We'll hear all the discussions of AV2. So it's gonna be really interesting and fascinating to follow. And we, we promise to to bring you all the updates here on The Video Insiders. So Pankaj I really want to thank you. This has been a fascinating discussion with very interesting insights into the world of codec development and compression and, and wavelets and DCT and and all of those topics and, and the history and the future. So thank you very much for joining us today on the video insiders.

Pankaj Topiwala: 51:25 It's been my pleasure, Mark and Dror. And I look forward to interacting in the future. Hope this is a useful for your audience. If I can give you a one parting thought, let me give this...

Pankaj Topiwala: 51:40 H.264 AVC was developed in 2003 and also 2004. That is you know, some 17 years or 16 years ago, it is close to being now nearly royalty-free itself. And if you look at the market share of video codecs currently being used in the market, for example, even in streaming AVC dominates that market completely. Even though VP8 and VP9 and VP10 were introduced and now AV1, none of those have any sizeable market share. AVC currently dominates from 70 to 80% of that marketplace right now. And it fully dominates broadcast where those other codecs are not even in play. And so they're 17, 16, 17 years later, it is now still the dominant codec even much over HEVC, which by the way is also taking an uptick in the last several years. So the standardized codecs developed by ITU and MPEG are not dead. They may just take a little longer to emerge as dominant forces.

Mark Donnigan: 52:51 That's a great parting thought. Thanks for sharing that. What an engaging episode Dror. Yeah. Yeah. Really interesting. I learned so much. I got a DCT primer. I mean, that in and of itself was a amazing,

Dror Gill: 53:08 Yeah. Yeah. Thank you.

Mark Donnigan: 53:11 Yeah, amazing Pankaj. Okay, well good. Well thanks again for listening to the video insiders, and as always, if you would like to come on this show, we would love to have you just send us an email. The email address is thevideoinsiders@beamr.com, and Dror or myself will follow up with you and we'd love to hear what you're doing. We're always interested in talking to video experts who are involved in really every area of video distribution. So it's not only encoding and not only codecs, whatever you're doing, tell us about it. And until next time what do we say Dror? Happy encoding! Thanks everyone.

In this episode, we catch up with the President & CEO of the company who won the 2018 Frost & Sullivan Global Content Protection Entrepreneurial Company of the Year Award, where we talk about DRM and its ever-changing role in the video industry.

Mark Donnigan: 00:00 This episode on DRM was so meaty, that we decided to jump right into a nine minute segment of Dror talking with Christopher Levy, who is President and CEO of BuyDRM, about how DRM technology fragmentation came to be, and the strategies behind DRM as a platform lock in. After this extended clip, we resumed the rest of the interview. You will definitely want to keep listening. Here's Dror and Christopher Levy.

Dror Gill: 00:35 This is really an interesting trend you're talking about. On one hand, you have these silos, and the silos include the software platforms, the hardware devices, the content, and the DRM mechanism, which is made by a certain, by a specific company. Now, some of these companies have interest only in parts of this type of ecosystem. For example, Samsung have devices, they have a software platform, they don't have their own DRM, and they don't have much content of their own. So, now this collaboration with Apple is bringing more content, a lot more content, to Samsung devices, and bringing a lot more devices to Apple's content. We all know, all of you know, the rumors about Apple expanding their content service to be much wider than it is today, so it really makes sense.

Dror Gill: 01:37 The topic you raise of which DRM will be used to enable this collaboration or cross-streaming of content between platform is really a very interesting issue. Another point you mentioned, which, you know, I can really resonate with, is the fact that standardization has happened across the video ecosystem in things, in components such as codex, packaging, controller mechanisms, manifests, things like that. And, DRM, although there have been attempts to standardize DRM, there has always been some internal component of that DRM that remained proprietary. That remained part of a closed, in siloed ecosystem such as PlayReady and Widevine, and this always struck me as kind of odd that everything else is standardized, and even the you know, mechanisms of exchanging keys in DRM's are defining DRM protocols.

Dror Gill: 03:08 Everything is standardized but finally, the key. Those very large companies do not want to give up the key. The key is what they control, and it is the key of opening the content, but also the key to the whole ecosystem, and platform which enables their own platforms to grow.

Dror Gill: 03:31 My question is, and referring to the fact that you also said that more and more layers or components of DRM are being standardized. Do you see somewhere in our near future that finally this content protection component will also be fully standardized, and in the same way that we're now having the harmonization of HLS and DASH with CMF, have harmonization of different DRM systems, and no single company would control those, this key to the industry?

Christopher Levy: 04:10 You make a really good point that, you know, in essence DRM and Codec have had similar kind of evolutions over time. If you look specifically at the DRM industry, and not to make a short story long, but to kind of paint a picture of why we're at, where we're at, you've got an odd mix of singularities that it would seem would leave almost no possibility that there would be a marketplace for DRM where their companies would have to pay for it, or that companies would continue to invest in it.

Christopher Levy: 04:46 I mean, if you fall way back to the beginning of the invention of DRM per se, as we know it, you fall way back to a meeting between Intertrust and Microsoft in, I think late 1999, where they agreed they were going to collaborate on some stuff. But then, at some point when Reciprocal launched, and decided that they were gonna partner close with Microsoft, Intertrust made an offer to Microsoft." Hey, give us two hundred and fifty million dollars, and license our technology," and a certain gentleman at Microsoft made the decision with his team to say, no. Only to later than lose a multi-billion dollar lawsuit to Intertrust, and Bill Gates wrote them a check that later allowed them then to go pursue every single company in the world that uses DRM. And so now, you've got Intertrust, who has a DRM, Marlin, that nobody uses in the U.S., only uses it in China, but Intertrust doesn't have a browser or an operating system. But, they own all the intellectual property around DRM, and so Apple, Microsoft, Google, Samsung, Sony, anyone in the world who touches DRM has had to take a license from Intertrust.

Christopher Levy: 06:00 But, then Intertrust, wasn't able to be successful with their own DRM technology, because, as I mentioned, they're locked out when it comes to having a browser or an operating system. So, they actually have somewhat abandoned Marlin, and moved to support Google, Apple, and Microsoft's DRM's. But then, you look at them and you say, "Okay, what would drive these companies to integrate such, so they an be interoperable?" Because that's kind of what we're talking about here, is how are Samsung and Apple gonna interop, but how is that gonna help everyone? Including HEVC, and what you find out is, that you know, DRM was clearly created. When I say created, when it was commercialized by Apple, Google, and Microsoft, it was obviously done on two kinda bifurcated paths.

Christopher Levy: 06:46 One, to satisfy potential looming lawsuits related to record labels, and studios, and artists, and creators, and content owners, pointing a finger at these large companies, saying your technology platforms are massive piracy platforms. Secondly, it was done as a platform play, to get you to use the platform. I mean if we look back at PlayReady. PlayReady was a technology that was completely driven to lock you in to using Windows based technologies, and Microsoft based technologies.

Christopher Levy: 07:16 Now, if you pull that out, if you pull Intertrust, and Microsoft completely out of the DRM discussion, and you just look at Apple and Google, who really are driving the entire industry now. They both have been using DRM to date, and on both those paths. To satisfy the lawyers, and to satisfy the lock in, and that is just where we're at, but, now the market has gotten so saturated.

Christopher Levy: 07:42 Google has not been successful selling devices. The Google Chromebook is a disaster. The Google Pixel phones are not selling as well as Google would expect they would sell, as the inventor, and owner of Android. So, now you get down to, okay, DRM previously was a legal thing, it was a lock in thing, but now, what is it? And I think what we're starting to see come to light is, that with the movement of common encryption, by you know, different various parties, the movement towards CMAF, the movement away ASCTR encryption, that was designed in PlayReady, into CBC encryption, we're really close to having a CMF, CMAF file, that using common encryption would have decryptors for Fairplay, PlayReady, and Widevine.

Christopher Levy: 08:37 So, we're getting very close to that. A deal like this, that Apple and Amazon have struck. It really could be the gas to the match. I sense that there's gonna be a push through here, the technology, Apple's Fairplay has gotten a lot of deployment experience now, so there's a big community contributing back to Apple.

Christopher Levy: 08:57 Apple has a very small team, if you knew the number of people working a DRM in Google and Apple, you would be shocked, and yet, they're converging. And, I think the reason they're converging, is that, you know, the consumer in the end, is dictating what they want, and consumers have made it very clear they want, you know, Samsung smart TV's. They want Apple TV's. They want Android tablets. They want Apple IPhones.

Christopher Levy: 09:24 I think both of them now, are gonna take a little play out of Steve Jobs DRM playbook, and probably find a way to cross pollinate their businesses, because Apple's not in the search business, you know. They try and interact in the home marketplace, but Google already owns the home, outside of Alexa. So, it's interesting, you know, to just clearly take one stab at it. I would say that we are headed towards complete inter op ability and that has a lot of benefits.

Christopher Levy: 09:57 It benefits operators, in cost reductions. It benefits consumers, in less confusion and playback stops. But mostly, it's gonna give Google a shot at, you know, exposing their offering to Apple's audience and vice versa.

Announcer: 10:15 The Video Insiders is the show that makes sense of all that is happening in the world of online video as seen through the eyes of a second generation Codec nerd, and a marketing guy who knows what Iframes and macro blocks are. Here are your hosts, Mark Donnigan, and Dror Gill.

Mark Donnigan: 10:36 Let's rejoin the interview with Christopher Levy from BuyDRM.

Christopher Levy: 10:41 To kind of just give a quick summary, the company is one of the dark horses of the content protection, and DRM business. We have a pretty well known brand as a company. We have extended our platform out pretty widely in the business. So we have a Multi-DRM platform called KeyOS, and we have a couple of different components of it.

Christopher Levy: 11:03 We have the encryption tools, we have the licensing tools, and we have the player tools, and we're integrated with about fifty different encoder server player companies in the marketplace. We service some of the major brands that you might be familiar with, like BBC iplayer BBC sounds, Sony Crackle, Showtime OTT, Blizzard, Warner Brothers, and we do a lot of work that we're not really at the liberty to discuss.

Christopher Levy: 11:30 But we do a lot of pre-release work as well. So, a lot of the focus in the business is on consumer media, but we also have a pretty significant business that's, you know, pre-release. So, Digital Daily, Screeners, Academy voters. We are very active in the Academy voter space. We currently host Apple Fairplay certificates for the five largest media companies in the world today. Some of which you're familiar with, I'm sure.

Christopher Levy: 11:57 To kind of fast forward, the company is privately owned. We are profitable. We own the company, myself, and the Chairman, Ron Baker, is my partner in the business, and we have different development teams based around the world. We've got our core team in Riga, Latvia. We have a team in Moscow, and a couple of people in St. Petersburg, and then we also have some people in Paris that work on our Android and IOS SDK's and our CTO is in Vancouver, and the company and myself, and the sales marketing management teams are all based in Austin, Texas, and yeah.

Christopher Levy: 12:34 Just to fast forward, we, late last year, for the first time ever, in the Frost and Sullivan Global Content Protection Report. This report is, you know, it's kind of a bigger picture report. It's kind of what they call content protection includes casts and DRM, so we are listed in the report with some of the heavy weights like you know NAGRA or Detto. But we were included in that report, and we ultimately were selected as the entrepreneurial company of the year for our variety of different business models.

Christopher Levy: 13:07 You know, we pride ourselves on having a very strong core DRM platform. But, we also now license our technology, so we've expanded into India, and all over Europe. We have several large major gaming companies, media companies that now run our software in their own data center, in their own cloud. So, that kind of vision shift in the company, I think is what got us over the goal line with the award.

Christopher Levy: 13:30 But, we're just you know, wrapping up one of our best years ever, if not our best year ever, last year. And, we're just kind of waiting to see all the different crazy announcements that come out of CES, you mentioned our team is there on site. But, I'm closely watching the announcements that Apple made about partnering with Samsung in LG, because it creates some very interesting possible synergies that all of us can benefit from.

Mark Donnigan: 13:56 Definitely. We're tracking that very closely as well. I mean, let's start there.

Christopher Levy: 14:03 Well, you know, the DRM industry at large is very interesting, because it has become a bit of the political third rail of digital media, as I'm sure you all know. At this point, each DRM technology is siloed into a global technology company. So, if you start left to right based on the kind of market, you know, availability of the product, you had Microsoft with PlayReady. PlayReady runs in IE and EDGE, and on Windows natively. You've got Google with Widevine that runs in Chrome, primarily on Windows and Android, but also runs in IOS. It's the one technology that runs on all three platforms, and you've got Apple's Apple Fairplay DRM, which really only works in Safari on MacOS, and Safari on IOS, and it works for tvOS. It will also work possibly on other products, we may find out here soon.

Christopher Levy: 15:11 I have to be careful what I say, but to kind of track what's going on, you know this announcement that Apple made about being able to move their business offering over to other platforms, I think, was largely driven by the tipping point of the iPhone sales over the past couple of years. It's no secret that Apple's last couple of iPhone product lines have not sold that well, so that's created kind of a tipping point in the company where now, they're trying to figure out, okay, where do we go next? And clearly Apple has a massive media empire.

Christopher Levy: 15:45 They're one of the first companies to ever have a license to just about every song, and movie, and TV show that consumers in America are familiar with. And, they obviously have a globally strong brand. But, because DRM has been a political silo, today, you know, iTunes doesn't appear on Android. It doesn't appear on Tizen. It's not on Ruku. It's not on Smart TV's. But, that is going to change, and the question is, how will it change?

Christopher Levy: 16:13 And to kind of give an example, if you take a look at Roku, who has gone through a similar transition where they were a streaming puck company, they were a streaming stick company. Then, Amazon entered the streaming stick company, and entered with Amazon prime, and Roku then suddenly decides, now, it's a content company. But, it also wants to get eyeballs and users onto it's platform, regardless of the direction it's going, and so Roku had to go. To support YouTube, they had to work with Google to implement Widevine DRM on the Roku platform, which previously was a PlayReady, and Verimatrix platform, natively and solely, and so that model where Roku kind of stepped over the fence and implemented Google's DRM to get YouTube is an interesting example of maybe what's going on with Samsung. We don't totally know yet. What Samsung and LG are doing, and we have our feelers out, and of course, we've talked to Apple pretty extensively about it, because we have a very close relationship with Apple as one of their frontline partners in the industry.

Christopher Levy: 17:18 But, I think it plays out one of two ways, and it is somewhat DRM dependent, and Codec dependent, because of the fact that Apple is either going to allow Samsung to distribute iTunes on their platform, or really Apple is gonna distribute, I say, because it's an open app marketplace. But, Apple has a decision to make, and it's, do they deploy it using WideVine, and reformat their application platform to use Widevine DRM instead of Fairplay, or does Samsung jump the shark and implement Fairplay?

Christopher Levy: 17:54 Because at the core of all these DRM's, the encryption decryption components are almost identical. At this point, all three DRM's us AES one twenty in encryption. There are some various different tweaks there, with regards to, the encryption mode CBC, verses CBR, but we're starting to see some standardization. I'm sure you're familiar with, with around formats. I personally believe it could go either way, or it could go both ways, because if Samsung were to implement Fairplay on their newer platforms, that would create a whole new synergy between Samsung and Apple that, oddly enough, hasn't been destroyed by the multi-billion dollar IP lawsuits that have gone back and forth between the two of them as vendors and competitors.

Christopher Levy: 18:40 But on the other hand, I could see, you know, Apple just wanting to push it out through Widevine, because if they got iTunes to work with Widevine, and I mean iTunes video is what I'm focused on, then the majority of the relatively, recently shipped Samsung TV's, more than likely, can all support iTunes. Which would be kind of cosmic shift in these siloed offerings that all fall back to DRM. Am I right? I mean, Apple's got iTunes on Fairplay. Google's got Google Play on Widevine. So, it's an interesting thing that's gonna happen. I am very curious myself.

Mark Donnigan: 19:21 It does sound like really good news ultimately. It's interesting your observation about, you know, the platform lock in. I'm thinking back to when I was active in the DECE, which became the ultraviolet, you know, which, was really revolutionary at the time. Because, you know, back then, you consume content from a particular store, if that was Vudu, for example. You were locked into Vudu, right. You know if Vudu wasn't on a particular device, then I was also locked in to the devices I could watch it on.

Mark Donnigan: 20:00 So,the consumer now is going to enjoy the benefit of this truly, any content, anywhere, on any device, at any time. You know, so, that's all very good things. You know Christopher, I was reading your blog and by the way, listeners should definitely go to the blog, why don't you tell them again, I just I don't recall the actual url. Tell them the address of your blog.

Christopher Levy: 20:29 Yeah, it's really simple to remember it’s: thedrmblog.com.

Mark Donnigan: 20:35 That's it. Thedrmblog.com that's awesome. Yeah, kinda like thevideoinsiders.com, that's right. No, Christopher, I want to get your comment on, I think it's your latest post, where you're talking about HTML5, kind of the App-less approach, and you know, I appreciated the article.

Mark Donnigan: 21:01 It was presenting a little bit of the pro's and the con's of, and I think you were doing it in the context of inflight entertainment. And, I know that people, if you're running a video service, if you're Amazon, if you're Netflix, you know, even if you're Vudu, Hulu, whatever, you know, they have to maintain up, hundred, you know, multiple hundreds of different player SDK's. You know, it's incredibly complex. So, the idea that you could perhaps, just scale that way back, and perhaps just go to an HTML5 app, is interesting. So, maybe you can share with the listeners, both, your thoughts, and the pro's and the con's, and give kind of a recap of that blog post.

Christopher Levy: 21:48 You bet. And, I mean clearly, that obviously, is also effected by the evolution of Codec, and HEVC and others, but there's this trend, and the in-flight entertainment space is an interesting creature. I've spent the past two years researching this space because previously BuyDRM had a bunch of clients in the space, but they were through third parties. So, you know, we had a business with Lufthansa, Technology Solutions, where they were deploying our technology in Virgin Airlines, LL Airlines, Lufthansa Airlines.

Christopher Levy: 22:24 They put the technology on Greyhound buses. Post Bus, which is the largest bus company in Germany, and we also have a little bit of business with companies like Global Eagle, and some others, and we started to look at, you know what's the opportunity for us to enter the space directly.

Christopher Levy: 22:41 So, we started going, attending shows, and doing research, talking to people. So, the way that in-flight entertainment systems make it on airplanes is different than you might expect. The airline industry has about four conglomerates that all, kind of, control what you call, you know, in-flight experience. Now, the in- flight experience, you know, the video piece is what we're focused on, but it includes interiors. It includes catering. It includes environment. It includes wifi. It includes being green. Entertainment's one component of it, but it's locked in with all these other kind of aspects of the business, and so therefore, it's treated in a very, what I would say, in a very institutional manner.

Christopher Levy: 23:19 To date, in-flight systems have been wired, and they're in your head rest, or it's a fold up screen if you're a business, and you're first class, it extends out a little booth you're in, and you're limited to watch videos that are in a dedicated platform that's hardwired on to the plane, and that was the experience.

Christopher Levy: 23:38 Then, along came satellite. Then, along came in-flight wifi. And, IFEC, you know, in-flight entertainment. The connected version with wires, suddenly pivoted to in- flight entertainment overnight. Which means wireless, and then DRM became a big topic. But, what you started to see DRM really drive, was the issue of, do airlines want to maintain premium content apps for their clients so they can watch content? Or do they just want them to open their browser, and get on the wifi network, sign in, and then have access to all the content through a browser?

Christopher Levy: 24:12 There's this trend in the business where a lot of companies have gone the direction of the browser, so like, if you get on a Southwest Airlines flight, you want to watch Dish TV live, you know, the implementation is there, on the plane. There's a dish receiver on the top of the plane that's got multiple different LMB's. Each channel is switchable. They got an encoder on the plane that takes the MPEG transport stream coming down over the dish, converts it, encrypts it, shoots it out of a server, on the plane to your browser. And that's easy, and it's fun. And it works, and it's especially effective for live TV.

Christopher Levy: 24:47 Stepping away from that, when you start to talk about doing things that are more efficient, and I think where consumers are headed, which is downloads, offline playback, bring your own device, the browser kind of starts to die because it doesn't work offline well. It doesn't do downloads well, because each browser has a protected limited amount of storage on the device it's running for security reasons. And browsers, the implementation, most players in them are not that efficient, and so what you find is that the browser is quicker, it's faster, it's dirtier, it's cheaper, but it opens up the door for a bunch of fails on the consumer side. Which is, decreased battery life, forced to use streaming, which uses the wifi radio which is decreased battery life, increased overhead on the aircraft.

Christopher Levy: 25:43 You don't get offline playback or download, so you can't download a stream and play it in a browser effectively offline. And lastly, consumers are very comfortable with their devices. Like, if I'm given the option of watching my ten point seven iPad pro with my bose qc thirty-five-two headphones, I'm gonna pick that every time over plugging some crappy, hand wiped headphones that hardly fit, that sound terrible, into a jack that's crackling, so I can watch a screen that has a four inch thick screen protector on it.

Christopher Levy: 26:20 The airline industries are trying to figure out, okay, well what do we do, because we're not OTT operators, but how do we make clients happy? And so, they're caught in a dilemma right now. Now, you know, I see it going two ways. I frankly think the live TV will continue to be in the browser. Remember, DRM adds some overhead cause you gotta decrypt the content and add some CPU overhead therefore decrease battery life.

Christopher Levy: 26:44 When you move to an app, I think apps are gonna be a lot more prevalent for VOD content and shifted viewing, and TV viewing. The last thing that's going on, that the airlines don't totally understand, and I've spent a lot of time trying to educate them about, and this is kind of a tangential issue, but I'm sure you can understand, is that every single passenger that's on an airplane, more than likely, and I said within a ninety percent or higher realm of operation, especially on International and business flights, have a Netflix, Itunes ,Google Play, Hulu account. And now, with DRM they can download all the movies and TV shows they want to their device, and just go on the plane, and have every blockbuster, every TV show, every highlight, every documentary, every podcast, that they want to. Access on their own device, and use it in their own way in their own time, in their own comfort.

Christopher Levy: 27:42 So, that's kind of the big divide right now, is companies are trying to figure out, well, we can save time and money on not having to build IFE apps, and just go to the browser, but we lose a bunch of things that consumers might want. There's a couple of other things which are also driving that, and those are accessibility issues which I think, will drive a lot of companies be forced to maintain apps, and those accessibility issues.

Christopher Levy: 28:03 Accessibility use on devices, you know, iPhone and Android have different functions for people that have disabilities or motor challenged, and aren't, you know, able to use the device the same way they would use an IFE platform where they gotta touch the screen in front of them, you know, reach up, and et cetera, et cetera.

Christopher Levy: 28:24 Secondly is multi-track audio. Thirdly, is multi-language caption support. I think those are the three issues, are more gracefully handled within applications.

Christopher Levy: 28:34 Lastly, I think applications are more likely to support advanced codecs, like HEVC, sooner. Because the applications are running on devices that are being modernized, updated, purchased more widely across a wider range of markets. And so, the people that design the player SDK's and apps, and the operating systems in the devices, are much more likely to embrace newer codecs like HEVC, then browser operators are. Because browsers update at a crawl.

Christopher Levy: 29:09 So, I mean, Google is the fastest browser updater in the business. But then, if you look at Safari, and IE and Edge, it's like, you know, waiting for your Grandmother to mail your birthday present. You get it like, four months later. But you're happy you got it, so I think that's the last kind of hidden thing, is that you know, within premium apps on devices, in a bring your own device model there's a greater chance that you're gonna get higher quality content sooner with DRM than you would in a set top box, or in a seat back implementation.

Mark Donnigan: 29:40 Yeah this is a really important discussion, I think for any of our listeners who are planning video services, and maybe, sort of haven't been able to do that next level of research and are thinking, "hey, you know, I can just appoint HTML five, it will reduce complexity, it will get me to market faster." Those are all true, but you have to know what you're also not gonna be able to deliver to your customer.

Mark Donnigan: 30:07 One of the other things, that I didn't hear you say, maybe I missed it, but I know one hundred percent, you know to be true, is that content licensing in some cases prohibits for example, HD in a browser, or certain browsers or in certain configurations. So yeah, you may be able to deliver in to that browser, but you're limited to SD, you know? 480p or maybe 720p, but not 1080p, so you're not able to deliver even the full quality.

Mark Donnigan: 30:41 Now, in-flight entertainment, the bandwidths are so low that you know, I think 1080p is not very common anyway, but the point is, is that those are even things that you have to think about.

Christopher Levy: 30:53 Well, a researcher David McCannon, he's pretty famous, he's a young guy over in the UK who previously, was responsible for a pretty significant kind of white hat hack that started to turn ugly. He's a pretty brilliant guy. He published some stuff on Monday of last week that indicated that he had breached Widevine's level three DRM. Which is the lowest level of DRM, mostly used in the Chrome browser, now it appears that what he breached, wasn't exactly Google's technology, but a third party[inaudible 00:31:30] technology that Google was using to wrap up their content decryption module that sits inside Chrome.

Christopher Levy: 31:37 But, it's a good example of where, devices, especially Android devices, you know, they have hardware in them that allows hardware assisted key management. So, they have a hardware manage black box that sits on the device that is basically impenetrable. And so, that's another benefit of using devices.

Christopher Levy: 32:02 Apple has the same thing, so Fairplay on IOS, taps into a trusted computing module that's on the chip that's in the iPhone or iPad. Same thing with Android with Google's DRM, you can get level one Widevine playback for HD, and 4K content on the device and then you can cast that out to a much bigger screen if you want over Chromecast, or over Airplay for example. So that's, that's another example where, you know, apps are much more secure than play back in the browser.

Christopher Levy: 32:34 So, what has to happen now is Google's gotta go modify, and what they're in the process of, from what I understand, of updating their content decryption module for Widevine and Chrome, so that their level three use, which is what most of the operators use, is safe.

Christopher Levy: 32:51 But again, they're operating on a non-native platform to them. Windows, in the most cases. Widevine also runs in Chrome on, on, on MacOS, but in those browser models, browsers are sitting on top of operating systems that the operator doesn't always own, and so that's again another benefit to using Premium apps.

Mark Donnigan: 33:15 This is an awesome lead in to a discussion about AV1, and DRM support. I don't know, have you had the chance to do some research around you know DRM support for AV1?

Christopher Levy: 33:30 Yeah, I mean, we've been following it pretty closely. We are really closely aligned with some companies that are working pretty seriously on it, I mean. We're very aligned with Google, and Bitmovin, and Amazon and Intel, and some of the other people that are involved in it.

Christopher Levy: 33:47 But again, the big question is, at what point does AV1 start to appear in content in browsers with DRM's? And I guess, the problem that we kinda have right now, is that that hasn't really happened, and they've done some kind of stuff playing around with Firefox, to play AV1 content. But really, it's gonna be up to, again, it's gonna be up to Apple, Google, and Microsoft. Right? Because they are the ones that own the DRM, and the browser, and so again, you've got a weird. It's not a simple economy of supply and demand, it's, there's this third, you know, Robby Botter's hidden hand that's influencing who is gonna implement what.

Christopher Levy: 34:43 You've got HEVC, which is widely deployed, heavily proven in the marketplace. It's gone through some royalty and licensing politics that are pretty consistent that all codecs go through. I kinda wish sometime, the encoding business had the same oligarchy god that DRM has, where Intertrust can just licensed everybody and be done with it.

Christopher Levy: 35:07 But, HEVC in comparison to AV1, in HEVC there are tons of documents on you know, Apple's developer page, Google's developer portal, Microsoft's developer portal. Showing how to use their DRM with HEVC on different platforms, and there are numerous, numerous gibset manufacturers as you well know, and which we provided you a list of, that support it. And also, its supported in a lot of the browsers already, if not all of them.

Christopher Levy: 35:36 AV1, on the other hand, is kind of nowhere with any of that yet, but it seems to be, you know, a little less encumbered with the intellectual property issues. But frankly, I feel like as it gets closer to being deployed, and people start to really get their hooks in to it, we're probably going to see the same thing happen to AV1 that happened to HEVC. But, I think it's gonna happen before it ever gets widely deployed, in my opinion.

Dror Gill: 36:06 You think, when you say the thing that's gonna happen are you referring to patent accusations or patent infringement?

Christopher Levy: 36:21 Yeah, I try not to pick a side, because you know let's face it. If you picked our entire industry, the two most researched dollar intensive things are codecs and DRM. You could build a Codec, and at the end of spending millions, throw it in the trash, because it didn't scale. You could build a DRM, and in the end, because you weren't doing a freedom to operate analysis ahead of time, find out that you built a great technology, but it's never gonna see the light of day in the market because you are infringing on someone else's IP.

Christopher Levy: 36:55 I think what's going on with HEVC is kind of normal, right? Like, all these companies invested in it. And clearly, they intend to see their return on the investment, and they're looking at what happened with H- two-sixty-four, the patent pull stuff, all the, the kind of facts that we all know that there's quite a few companies in the business that aren't reporting royalties properly, and have kind of jumped the shark there.

Christopher Levy: 37:20 So, I think HEVC has a better chance than AV1, if I were to weigh the two. Just because, it's, you know, all the points I've mentioned; much further widely deployed, chip support, browser support, DRM support. AV1 doesn't have any of that, and it doesn't have the encumbrances of potential legal battles, yet. But, I don't know. What do you guys think is gonna happen, when it comes time to walk the aisle with AV1?

Dror Gill: 37:46 Indeed, nobody is giving you identification against any patent lawsuits for AV1. The companies involved in developing the codec itself, have signed agreements that they will not switch other, or the users of AV1, but this doesn't mean that somebody else will not claim any IP rights on algorithms used in AV1.

Dror Gill: 38:18 And, on the other hand, the conclusion that we reached is that, the fact, it is well know that AV1, right now, is much more computationally complex then HEVC. Right now, it's like a hundred times more complex, and even the people involved in AV1 development have told us that in the end, when everything is optimized, it will still be five to ten time more complex than HEVC.

Dror Gill: 38:49 And, we think that one of the reasons for that is all of that side stepping of patents. All of these techniques, which to be efficient in terms of bitrate consumption, as HEVC, but cannot use the same tools, and therefore I have to go in very weird ways around those protected methods in order to achieve the same result. And this is part of the problem, and why it is so computationally complex.

Dror Gill: 39:26 Recently, I've come up with yet another conspiracy theory after hearing that a lot of the decisions made, somebody wrote this in a blog post. A lot of the decisions that were made during the development of AV1 were driven by the hardware companies were members of the AOM.

Christopher Levy: 39:47 I was just gonna say that, Dror, is that A. There's not free lunch whether it's physics, mathematics, which is you know, part of physics in technology, in relationships, in religion, and that doesn't surprise me.

Christopher Levy: 40:02 But, what I was gonna point out, was Occam's razor says, "the simplest answer is more than likely, the answer," is correct. I would say, that is what's driving it, because let's face it, I mean, there's not a person working on it that doesn't benefit from that. I'm pretty sure that Google, Amazon, Microsoft, Apple, all the other companies sell computing software, and technology and silicon, and intel, so I can't imagine why that wouldn't be the case.

Christopher Levy: 40:32 But, you make a good point, that regardless of the fact that their trying to ignore the three laws of thermodynamics. I imagine they are, have a strategy for how they're going to sort that out, but the question is, will it really work? And, the only thing too, is if they don't adopt DRM into their message share pretty soon, and start showing examples of AV1 content with DRM, it's just gonna be another Ultraviolet. It's gonna be shiny. It's gonna sparkle. It's gonna have all the right looks and feels. It's got a cool logo. The stuff on the side is really cool, but will people use it, or is it just gonna be another augmented reality, virtual reality three-d, a year from now?

Mark Donnigan: 41:14 You know, I sometimes find myself feeling a little agitated or sitting in a conference, and I'm listening to a panel, and I'm hearing either a panelist or even Mozilla, you know, saying, you know," it’s coming, player support is coming. It's just months away. It's gonna be in the browser." And then they start, and I'm going, so, really? So, Sony pictures, and Warner Brothers is gonna allow you to play their movies inside a browser without DRM? Yeah. Let's see how that works. You know? Like? Then you've got up on the stage usually, or you hear speakers, and they're throwing off big service names, and Netflix is heavily behind AV1, so I am not naïve that Netflix is having these discussions, I'm sure.

Mark Donnigan: 42:04 But the point is, that DRM is DRM. It has to be implemented. It has to, to work with the standards the content owners accept. But the fact that you don't hear DRM, it’s sort of just... it's almost like, oh yeah, yeah. It's gonna be in the browser. It's gonna be supported. I'm like, that's just not how it works. It will come later. It's coming, don't worry about it.[inaudible 00:42:29]

Christopher Levy: 42:28 I mean, nevermind the battle that was fought at the W3C by all the media companies just named, and a hundred more, along with Google, and Apple, and Microsoft to implement DRM in the browser, because they know that's where people want to view content on their computers, whether it be desktop or laptop.

Christopher Levy: 42:46 But, they didn't do all the work, and engineering to get MSC and CDM's working to just all of the sudden, say, "see we're gonna throw it out the window because there's this new Codec in town."

Mark Donnigan: 43:00 Yeah, exactly. Exactly. So, wow. Well, I'm looking at our time here. This has been an amazing discussion Christopher, and we absolutely, need to have you back because we didn't get to talk about players, and I know you guys are active, and I know also in the player development. So, I think, Dror, what do you think? I think a part two should be players.

Dror Gill: 43:24 You know, Mark, Christopher did make this analogy between Codecs and DRM in one of the first episodes we told. Like the story of the Codecs, how they've been developed and DRM is also really a fascinating story, and even more because it's beyond standards that spam dozens of companies. It's really a few companies holding the power, holding the key, and that's also the DRM key. In the whole industry, and how it's gonna develop in the future I think would be really interesting to see whether we are going for true standards, finally, and a much easier life for consumers to play their content anywhere, or do we still have few years of struggling? So, really, thank you very much, Christopher.

Mark Donnigan: 44:16 Christopher, your website is Buydrm.com, correct?

Christopher Levy: 44:23 That's correct, and the blog is thedrmblog.com and once you guys get this podcast up and done, we'll go ahead and feature it on the blog, and I just wanted to quickly mention that in the next couple of days, we're gonna have a new blog post come out about deploying secure SDK's. And, we tackle a lot of the issues we talked about here in a generalist way. We do talk about our own SDK players, but I'll notify you when that blog is up. I think your readers will find it interesting.

Christopher Levy: 44:53 We're also have an HEVC update on our blog, but after today once you post the final edited blog, then we'll go ahead and roll out our update that I provided you with regards to kinda where the market's at as well.

Mark Donnigan: 45:09 Awesome. Awesome. Sounds good. Okay, well, we want to thank you again for listening to this incredibly engaging episode of the video insiders and until next time, what do we say Dror? Encode on? Is that our new..?

Dror Gill: 45:29 Encode on! Encode happily!

Mark Donnigan: 45:30 Encode happily, we've got to come up with something.

Dror Gill: 45:32 Yeah, we need to invent something like, you can never compress too much.

Mark Donnigan: 45:36 That's right, you can never compress too much, but you must preserve all the original quality. Alright, have a great day everyone. Thank you for listening.

Christopher Levy: 45:45 Thank you

Announcer: 45:47 Thank you for listening to the Video Insiders podcast. A production of Beamer limited. To begin using Beamer's Codecs today, go to Beamer.com/free to receive up to one hundred hours of no cost HEVC and H.264 transcoding every month.

Open source codec pioneer, Tom Vaughan, talks about the advantages & disadvantages of proprietary & open source technology. What he says may surprise you – despite which side of the fence you are on.

The following blog post first appeared on the Beamr blog at: https://blog.beamr.com/2019/01/24/in-the-battle-between-open-source-proprietary-technology-does-video-win-podcast/

Video engineers dedicated to engineering encoding technologies are highly skilled and hyper-focused on developing the foundation for future online media content. Such a limited pool of experts in this field creates a lot of opportunity for growth and development, it also means there must be a level of camaraderie and cooperation between different methodologies.

In past episodes, you’ve seen The Video Insiders compare codecs head-to-head and debate over their strengths and weaknesses. Today, they are tackling a deeper debate between encoding experts: the advantages and disadvantages of proprietary technology vs. community-driven open source.

In Episode 05, Tom Vaughan surprises The Video Insiders as he talks through his take on open source vs. proprietary technology.

Press play to hear a snippet from Episode 05, or click here for the full episode.
Want to join the conversation? Reach out to TheVideoInsiders@beamr.com

TRANSCRIPTION (lightly edited to improve readability only)

Mark Donnigan: 00:00 In this episode, we talk with a video pioneer who drove a popular open source codec project before joining a commercial codec company. Trust me, you want to hear what he told us about proprietary technology, open source, IP licensing, and royalties.

Announcer: 00:18 The Video Insiders is the show that makes sense of all that is happening in the world of online video, as seen through the eyes of a second generation codec nerd and a marketing guy who knows what iframes and macroblocks are. Here are your hosts, Mark Donnigan and Dror Gill.

Mark Donnigan: 00:35 Okay.

Mark Donnigan: 00:35 Well, welcome back everyone to this very special edition. Every edition is special, isn’t it, Dror?

Dror Gill: 00:43 That’s right. Especially the first editions where everybody’s so excited to see what’s going to happen and how it would evolve.

Mark Donnigan: 00:49 You know what’s amazing, Dror, we had in the first 48 hours, more than 180 download.

Dror Gill: 00:55 Wow.

Mark Donnigan: 00:56 You know, we’re like encoding geeks. I mean, are there even 180 of us in the world?

Dror Gill: 01:01 I don’t know. I think you should count the number of people who come to Ben Wagoner’s compressionist breakfast at NAB, that’s about the whole industry, right?

Mark Donnigan: 01:09 Yeah. That’s the whole industry.

Mark Donnigan: 01:11 Hey, we want to thank, seriously in all seriousness, all the listeners who have been supporting us and we just really appreciate it. We have an amazing guest lined up for today. This is a little personal for me. It was IBC 2017, I had said something about a product that he was representing, driving, developing at the time. In fact, it was factually true. He didn’t like it so much and we exchanged some words. Here’s the ironic thing, this guy now works for us. Isn’t that amazing, Dror?

Click to view x265 vs. Beamr 5 speed and performance test.

Dror Gill: 01:49 Yeah, isn’t that amazing?

Mark Donnigan: 01:52 You know what, and we love each other. The story ended well, talk about a good Hollywood ending.

Mark Donnigan: 01:58 Well, we are talking today with Tom Vaughn. I’m going to let you introduce yourself. Tell the listeners about yourself.

Tom Vaughn: 02:10 Hey Mark, hey Dror. Good to be here.

Tom Vaughn: 02:12 As Mark mentioned, I’m Beamr’s VP of strategy. Joined Beamr in January this year. Before that I was Beamr’s, probably, primary competitor, the person who started and led the x265 project at MulticoreWare. We were fierce competitors, but we were always friendly and always friends. Got to know the Beamr team when Beamr first brought their image compression science from the photo industry to the video industry, which was three or four years ago. Really enjoyed collaborating with them and brainstorming and working with them, and we’ve always been allies in the fight to make new formats successful and deal with some of the structural issues in the industry.

Dror Gill: 03:02 Let me translate. New formats, that means HEVC. Structural issues, that means patent royalties.

Tom Vaughn: 03:08 Yes.

Dror Gill: 03:09 Okay, you can continue.

Tom Vaughn: 03:11 No need to be subtle here.

Tom Vaughn: 03:13 Yeah, we had many discussions over the years about how to deal with the challenging macro environment in the codec space. I decided to join the winning team at Beamr this year, and it’s been fantastic.

Mark Donnigan: 03:28 Well, we’re so happy to have you aboard, Tom.

Mark Donnigan: 03:32 I’d like to just really jump in. You have a lot of expertise in the area of open source, and in the industry, there’s a lot of discussion and debate, and some would even say there’s religion, around open source versus proprietary technology, but you’ve been on both sides and I’d really like to jump into the conversation and have you give us a real quick primer as to what is open source.

Tom Vaughn: 04:01 Well, open source is kind of basic what it says is that you can get the full source code to that software. Now, there isn’t just one flavor of open source in terms of the software license that you get, there are many different open source licenses. Some have more restrictions and some have less restrictions on what you can do. There are some well known open source software programs and platforms, Linux is probably the most well known in the multimedia space, there’s FFmpeg and Libav. There’s VLC, the multimedia player. In the codec space, x264, x265, VP9, AV1, et cetera.

Dror Gill: 04:50 I think the main attraction of open source, I think, the main feature is that people from all over the world join together, collaborate, each one contributes their own piece, then somehow this is managed together. Every bug that is discovered, anyone can fix it, because the source is open. This creates kind of a community and together a piece of software is created that is much larger and more robust than anything that a single developer could do on his own.

Tom Vaughn: 05:23 Yeah, ideally the fact that the source code is open means that you have many sets of eyes, not only trying the program, but able to go through the source code and see exactly how it was written and therefore more code review can happen. On the collaboration side, you’re looking for volunteers, and if you can find and energize many, many people worldwide to become enthusiastic and devote time or get their companies motivated to allocate developers full- or part-time to a particular open source project, you get that collaboration from many different types of people with different individual use cases and motivations. There are patches submitted from many different people, but someone has to decide, does that patch get committed or are there problems with that? Should it be changed?

Tom Vaughn: 06:17 Designed by a committee isn’t always the optimal, so someone or some small group has to decide what should be included, what should be left out.

Dror Gill: 06:27 It’s interesting to see, actually, the difference between x264 and x265 in this respect, because x264, the open source implementation of x264 was led by a group of developers, really independent developers, and no single company was owning or leading the development of that open source project. However, with x265, which is the open source implementation of HEVC, your previous company, MulticoreWare, has taken the lead and devoted, I assume, most of the development resources that have gone into the open source development, most of the contributions came from that company, but it is still an open source project.

Tom Vaughn: 07:06 That’s right. x264 was started by some students at a French university, and when they were graduating, leaving the university, they convinced the university to enable them to take the code with them, essentially under an open source license. It was very much grassroots open source beginnings and execution where developers may come and go, but it was a community collaboration.

Tom Vaughn: 07:31 I started x265 at MulticoreWare with a couple of other individuals, and the way we started it was finding some commercial companies who expressed a strong interest in such a thing coming to life and who were early backers commercially. It was quite different. Then, because there’s a small team of full-time developers on it working 40 hours plus a week, that team is moving very fast, it’s organized, it’s within a company. There was less of a need for a community. While we did everything we could to attract more external contributors, attracting contributors is always a challenge of open source projects.

Mark Donnigan: 08:14 What I hear you saying, Tom, is it sounds like compared to the x264 project, the x265 project didn’t have as large of a independent group of contributors. Is that …?

Tom Vaughn: 08:29 Well, x264 was all independent contributors.

Mark Donnigan: 08:32 That’s right.

Tom Vaughn: 08:33 And still is, essentially. There are many companies that fund x264 developers explicitly. Chip companies will fund individual developers to optimize popular open source software projects for their instruction set. AVX, AVX2, AVX512, essentially, things like that.

Tom Vaughn: 08:58 HEVC is significantly more complex than AVC, and I think, if I recall correctly, x265 already has three times the number of commits than x264, even though it’s only been in existence for one third of the life.

Dror Gill: 09:12 So Tom, what’s interesting to me is everybody’s talking about open source software being almost synonymous with free software. Is open source really free? Is it the same?

Tom Vaughn: 09:23 It can be at times. One part depends on the license and the other part depends on how you’re using the software. For example, if it’s a very open license like Apache, or BSD, or UIUC, that’s an attribution only license, and you’re pretty much free to create modifications, incorporate the software in your own works and distribute the resulting system.

Tom Vaughn: 09:49 Software programs like x264 and x265 are licensed under the GNU GPL V2, that is an open source license that has a copyleft requirement. That means if you incorporate that in a larger work and distribute that larger work, you have to open source not only your modifications, but you have to open source the larger work. Most commercial companies don’t want to incorporate some open source software in their commercial product, and then have to open source the commercial product. The owners of the copyright of the GPL V2 code, x264 LLC or MulticoreWare, also offer a commercial license, meaning you get access to that software, not under the GNU GPL V2, but under a separate, different license, in which case for you, it’s not open source anymore. Your commercial license dictates what you can and can’t do. Generally that commercial license doesn’t include the copyleft requirement, so you can incorporate it in some commercial product and distribute that commercial product without open sourcing your commercial product.

Dror Gill: 10:54 Then you’re actually licensing that software as you would license it from a commercial company.

Tom Vaughn: 10:59 Exactly. In that case it’s not open source at all, it’s a commercial license.

Dror Gill: 11:04 It’s interesting what you said about the GPL, the fact that anything that you compile with it, create derivatives of, incorporate into your software, you need to open source those components that you integrate with as well. I think this is what triggered Steve Ballmer to say in 2001, he said something like, “Open source is a cancer that spreads throughout your company and eats your IP.” That was very interesting. I think he meant mostly GPL because of that requirement, but the interesting thing is that he said that in 2001, and in 2016 in an interview, he said, “I was wrong and I really love Linux.” Today Microsoft itself open sources a lot of its own development.

Mark Donnigan: 11:48 That’s right. Yeah, that’s right.

Mark Donnigan: 11:50 Well Tom, let’s … This has been an awesome discussion. Let’s bring it to a conclusion. When is proprietary technology the right choice and when is open source maybe the correct choice? Can you give the listeners some guidelines?

Tom Vaughn: 12:08 Sure, people are trying to solve problems. Engineers, companies are trying to build products and services, and they have to compete in their own business environment. Let’s say you’re a video service and you run a video business. The quality of that video and the efficiency that you can deliver that video matters a lot. We know what those advantages of open source are, and all things being equal, people gravitate towards open source a lot because engineers feel comfortable actually seeing the source code, being able to read through it, find bugs themselves if pushed to the limit.

Tom Vaughn: 12:45 At the end of the day, if an open source project can’t produce the winning implementation of something, you shouldn’t necessarily use it just because it’s open source. At the end of the day you have a business to run and what you want is the most performant libraries and platforms to build your business around. If you find that a proprietary implementation in the long run is more cost effective, more efficient, higher performance, and the company that is behind that proprietary implementation is solid and is going to be there for you and provide a contractual commitment to support you, there’s no reason to not choose some proprietary code to incorporate into your product or service.

Tom Vaughn: 13:32 When we’re talking about codecs, there are particular qualities I’m looking for, performance, how fast does it run? How efficiently does it utilize compute resources? How many cores do I need in my server to run this in real time? And compression efficiency, what kind of video quality can I get at a given bit rate under a given set of conditions? I don’t want the second best implementation, I want the best implementation of that standard, because at scale, I can save a lot of money if I have a more efficient implementation of that standard.

Mark Donnigan: 14:01 Those are excellent pointers. It just really comes back to we’re solving problems, right? It’s easy to get sucked into religious debates about some of these things, but at the end of the day we all have an obligation to do what’s right and what’s best for our companies, which includes selecting the best technology, what is going to do the best job at solving the problems.

Mark Donnigan: 14:24 Thank you again for joining us.

Tom Vaughn: 14:25 My pleasure, thank you.

Dror Gill: 14:26 I would also like to thank you for joining us, not only joining us on this podcast, but also joining Beamr.

Mark Donnigan: 14:32 Absolutely.

Mark Donnigan: 14:33 Well, we want to thank you the listener for, again, joining The Video Insiders. We hope you will subscribe. You can go to thevideoinsiders.com and you can stream from your browser, you can subscribe on iTunes. We’re on Spotify. We are on Google Play. We’re expanding every day.

Announcer: 14:57 Thank you for listening to The Video Insiders podcast, a production of Beamr Limited. To begin using Beamr’s codecs today, go to Beamr.com/free to receive up to 100 hours of no-cost HEVC and H.264 transcoding every month.