Sunday, 5 August 2018

Consuming video from WCF as a Base64 encoded strings

This article follows up our quest of loading up video byte arrays from WCF efficiently, but now the byte array is converted into a Base64 encoded string. You can start by cloning the repository I have prepared from here:
git clone https://toreaurstad@bitbucket.org/toreaurstad/wcfaudiostreamdemo.git git fetch && git checkout VideBase64String_05082018
Base64 encoded strings take six bits from the byte array and designates is as a char where the char can be one of 64 characters, in MIME implementation it is [A-z][0-9] and + and /, which is 64 different characters that the six bytes are encoded into. This means that 24 bits can be represented as three Base64 encodeded characters, that is 3 bytes in our byte array can be represented as FOUR base64 encoded chars. This 3:4 ratio is the rationale behind the MTOM optimization in WCF (to be discussed later). However, recap from the previous article where we downloaded a video sizing 4 MB on disk and Fiddler reporting it to be about 3 times larger. This is bloated data that I want to explore if we can fix up a bit. The bloated data is because I have managed not to truly enforce the XMLHttpRequest to send the data through the WCF REST binding (webHttpBinding) as true binary data - binary data is still sent as a string object to Javascript. This is sad, and I want to try to speed this up and avoid bloated data. To get this size down, we can start by using a Base64 encoded string instead in our XmlHttpRequest. First off, we define a service contract operation to return our string with Base64 data:
        [OperationContract]
        [WebGet(UriTemplate = "mediabytes/{videofile}")]
        string GetVideoAsBase64(string videofile);
Now our Service Implementation looks like this:

using System;
using System.IO;
using System.ServiceModel.Activation;
using System.ServiceModel.Web;
using System.Web;
using WcfStreamAudioDemo.Common;

namespace WcfStreamAudioDemo.Host
{

    [AspNetCompatibilityRequirements (RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
    public class VideoService : IVideoServiceContract
    {

        public Stream GetVideo(string videofile)
        {
            return GetVideoStream(videofile, ".mp4");
        }

        public string GetVideoAsBase64(string videofile)
        {
            Stream videoStream = GetVideoStream(videofile, ".mp4");

            byte[] buffer = new byte[32*1024];
            int count;
            var memoryStream = new MemoryStream();

            while ((count = videoStream.Read(buffer, 0, buffer.Length)) > 0)
            {
                memoryStream.Write(buffer, 0, count);
            }

            return Convert.ToBase64String(memoryStream.ToArray()); 
        }

        private Stream GetVideoStream(string videofile, string extension)
        {
            string mediaFolder = HttpContext.Current.Server.MapPath("~/media");
            string videoFileFullPath = Path.Combine(mediaFolder, videofile) + extension;
            return File.OpenRead(videoFileFullPath);
        }

    }
}


The client side code will now retrieve a Base64Encoded string. The following client side scripts loads up our video:

 <script type="text/javascript">

        window.onload = init;
      
        var source; //Video buffer 

        function init() {
            loadByteArray('http://myserver/WcfStreamAudioDemo.Host/VideoService.svc/mediabytes/sintel_trailer-480p');
        }

        function base64ToArrayBuffer(base64) {
            var binaryString =  window.atob(base64);
            var len = binaryString.length;
            var bytes = new Uint8Array( len );
            for (var i = 0; i < len; i++)        {
                bytes[i] = binaryString.charCodeAt(i);
            }
            return bytes.buffer;
        }

        function loadByteArray(url) {
            
            var request = new XMLHttpRequest();

            //request.overrideMimeType('text\/plain; charset=x-user-defined');
            //request.responseType = 'blob';

            request.open('GET', url, true);
            //request.setRequestHeader("Content-Type", "video/mp4");


            request.onload = function() {
                //console.log(request.response);
                debugger;

                var responseData = request.response;
                if (responseData === undefined || responseData === null)
                    return;

                responseData = responseData.slice(0, -9);
                responseData = responseData.substring(68);

                var videoByteArrayView = base64ToArrayBuffer(responseData);
            
                var blob = new Blob([videoByteArrayView], { type: "text/plain;charset=utf-8" });

                var blobUrl = URL.createObjectURL(blob);

                var videoCtrlFedByByteArray = document.getElementById("videoCtrlFedByByteArray");

                videoCtrlFedByByteArray.setAttribute("src", blobUrl);


            } //request.onload 

            request.send();

        }

    </script>

There is still a lot of manual Js scripting here to prepare the video here, I will try to explain. We have now managed to drastically reduce the bloated data of our original 4.4 MB video down from 15+ MB to just 5.8 MB using Base64 encoded string. The retrieved Base64 encoded string is first decoded with the Js atob() function ("ASCII to BINARY"). A Uint8Array is initialized with the length of this decoded binary string and the binary string is iterated by using the charCodeAt method, returning the ArrayBuffer inside the constructed Uint8Array. Note that I got the Base64 encoded string as a XML first, which is why I do some chopping off the string using slice and substring. Anyways, after getting an ArrayBuffer from the Uint8Array's buffer property, we can construct a Blob object (Binary large object) and create an object url and then set the "src" attribute of our HTML5 Video control to this blobUrl. This is still not very elegant, I have seen examples using "arraybuffer" as responseType of the XmlHttpRequest object in Js to retrieve binary data, but I got garbled data back trying to use it with WCF REST. So in this article you have seen a way to send binary data from a WCF method by using a Base64 encoded string as an intermediary between the Server and the client. We got the file size of the request according to Fiddler inspection down to much less than the bloated binary array transfer that actually sent the byte array as a text string back to the client.

Wcf byte array + HTML 5 Video + Custombinding

Again, this article looks at different ways to load byte array from WCF representing video data to web clients, this approach will return a byte array and use a CustomBinding in WCF. You can start by cloning the repository I have prepared from here:
git clone https://toreaurstad@bitbucket.org/toreaurstad/wcfaudiostreamdemo.git git fetch && git checkout VideBase64String_05082018
The following method is added to our WCF service contract, note that now we leave the webHttpBinding and use a CustomBinding.

     [OperationContract]
     byte[] GetVideoBytes(string videofile);

There is no [WebGet] attribute this time, as noted we will not use WCF REST but SOAP instead. The web.config of the host website for the WCF services exposes this CustomBinding:

<system.serviceModel>

    <services>
      
      <service name="WcfStreamAudioDemo.Host.AudioService">
        <endpoint behaviorConfiguration="RestBehaviorConfig" binding="webHttpBinding" bindingConfiguration="HttpStreaming" contract="WcfStreamAudioDemo.Common.IAudioServiceContract" />
      </service>
      
      <service name="WcfStreamAudioDemo.Host.VideoService">
        <endpoint behaviorConfiguration="RestBehaviorConfig" binding="webHttpBinding" bindingConfiguration="HttpStreaming" contract="WcfStreamAudioDemo.Common.IVideoServiceContract" />
        <endpoint address="custom" binding="customBinding" bindingConfiguration="CustomBinding" contract="WcfStreamAudioDemo.Common.IVideoServiceContract" />
      </service> 

    </services>
    

    <bindings>
      
     <customBinding>
        <binding name="CustomBinding">
          <binaryMessageEncoding>
            <readerQuotas maxArrayLength="100000000" maxStringContentLength="100000000"/>
          </binaryMessageEncoding>
          <httpTransport />
        </binding>
      </customBinding>

      <webHttpBinding>
        <binding name="HttpStreaming" transferMode="Streamed" maxReceivedMessageSize="1000000000" maxBufferPoolSize="100000000">
          <readerQuotas maxArrayLength="100000000" maxStringContentLength="100000000"/>
        </binding>
      </webHttpBinding>
    </bindings>
    
    <behaviors>

      <serviceBehaviors>
        <behavior>
          <!-- To avoid disclosing metadata information, set the values below to false before deployment -->
          <serviceMetadata httpGetEnabled="true" httpsGetEnabled="true"/>
          <!-- To receive exception details in faults for debugging purposes, set the value below to true.  Set to false before deployment to avoid disclosing exception information -->
          <serviceDebug includeExceptionDetailInFaults="true"/>
        </behavior>
      </serviceBehaviors>
     
      <endpointBehaviors>
        <behavior name="RestBehaviorConfig">
          <webHttp />
        </behavior>
      </endpointBehaviors>

    </behaviors>
    <protocolMapping>
        <add binding="basicHttpsBinding" scheme="https" />
    </protocolMapping>    
    <serviceHostingEnvironment aspNetCompatibilityEnabled="true" multipleSiteBindingsEnabled="true" />
  </system.serviceModel>

Note that the custom binding allows us to set up a binaryMessageEncoding. The next step is to go to the client project and add a service reference to the host project containing the WCF service. The app.config file is then updated, relevant parts shown here:
 <system.webServer>
    <directoryBrowse enabled="true" />
    <security>
      <requestFiltering>
        <requestLimits maxAllowedContentLength="1073741824" />
      </requestFiltering>
    </security>
        <staticContent>
            <mimeMap fileExtension=".mp4" mimeType="video/mp4" />
        </staticContent>
  </system.webServer>


  <system.serviceModel>
    <bindings>
      
      <customBinding>
        <binding name="CustomBinding">
          <binaryMessageEncoding>
            <readerQuotas maxArrayLength="100000000" maxStringContentLength="100000000" />
          </binaryMessageEncoding>
          <httpTransport maxReceivedMessageSize="100000000" />
        </binding>
      </customBinding>

    </bindings>
    <client>
      <endpoint address="http://he139920.helsemn.no/WcfStreamAudioDemo.Host/VideoService.svc/custom" contract="VideoService.IVideoServiceContract" binding="customBinding" bindingConfiguration="CustomBinding" />
    </client>
    
  </system.serviceModel>
Note the changes of setting up a MIME map for .mp4 files and setting up requestlimits. I will explain this soon. The client script now actually consists both of a server side script run in ASP.NET that sets up the video control in two ways. One way is to construct a HTML5 Data Url. This clearly was demanding for the browser to cope with and is not recommended. The video is actually embed on the page as a base64 encoded string! For a small video such as our video, it is actually possible still. It is of course fascinating that we can embed an entire video on our ASPX web page just as a Base64 encoded string, like it or not. The way to do this anyways is like this:

<script runat="server">

    private void btnInvokeWcfService_OnClick(object sender, EventArgs e)
    {

        using (var proxy = new VideoServiceContractClient())
        {
            byte[] payload = proxy.GetVideoBytes("sintel_trailer-480p");

            SetVideoSourceToHtmlDataUri(payload);

        }
    }


    private void SetVideoSourceToHtmlDataUri(byte[] payload)
    {
        //Set a base64 Html data uri

        videoCtrlFedByByteArrayThroughProxy.Attributes["type"] = "video/mp4";

        string base64String = Convert.ToBase64String(payload);

        videoCtrlFedByByteArrayThroughProxy.Attributes["src"] = "data:video/mp4;base64," + base64String;
    }

</script>

The following image shows how this actually works! Of course, this gives rendering and performance issues, as the web page now got very large content - the entire video is embedded into the page! Another way is to write the byte array to a temporary file on the server, and set the src attribute to this temporary file.

<script runat="server">

    private void btnInvokeWcfService_OnClick(object sender, EventArgs e)
    {

        using (var proxy = new VideoServiceContractClient())
        {
            byte[] payload = proxy.GetVideoBytes("sintel_trailer-480p");

            SetVideoSourceToTempFile(payload);

        }
    }

    private void SetVideoSourceToTempFile(byte[] payload)
    {
        //write to a temp file 
        string tempfile = Path.GetRandomFileName() + ".mp4";
        string tempfilePathForWebServer = HttpContext.Current.Server.MapPath("media/") + tempfile;
        File.WriteAllBytes(tempfilePathForWebServer, payload);

        videoCtrlFedByByteArrayThroughProxy.Attributes["src"] = "media/" + tempfile;
    }

</script>

Note that in these two samples, I have adjusted the HTML5 video control to be accessible to ASP.NET like this:

   <asp:Button runat="server" ID="btnInvokeWcfService" Text="Load Video" OnClick="btnInvokeWcfService_OnClick"/>
        
        <video id="videoCtrlFedByByteArrayThroughProxy" type="video/mp4"  runat="server"  controls width="320" height="240">         
            <p>Your browser doesn't support HTML5 video. Here is a <a href="http://wcfaudiodemohost.azurewebsites.net/VideoService.svc/mediabytes/sintel_trailer-480p">link to the video</a> instead.</p> 
        </video>


This method of retrieving video from WCF is the quickest way, it only fetches the original byte array data and it loads quickly. An adjusted version would not write to a file directly accessible on the web server, but use for example IsolatedStorage instead. I will look into that in the future. Hope you found this article interesting. CustomBindings in WCF gives you a lot of flexibility!

Loading video from WCF into HTML5 Video

Live demo here!

WCF demo - Loading video using Stream or byte array into HTML5 Video control
This article will look at loading up a video into a HTML5 Video. First off, you can clone the repository I have prepared here:
git clone https://toreaurstad@bitbucket.org/toreaurstad/wcfaudiostreamdemo.git
git fetch && git checkout VideoDemo_04082018


The image below shows our user interface, a simple web page displaying two HTML5 Video controls. This article builds upon code from previous articles. Defining a WCF Service Contract IVideoServiceContract is as following:

using System.IO;
using System.ServiceModel;
using System.ServiceModel.Web;

namespace WcfStreamAudioDemo.Common
{
    
    [ServiceContract]
    public interface IVideoServiceContract
    {

        [OperationContract]
        [ContentType("video/mp4")]
        [WebGet(UriTemplate = "media/{videofile}")]
        Stream GetVideo(string videofile);

        [OperationContract]
        [WebGet(UriTemplate = "mediabytes/{videofile}", ResponseFormat = WebMessageFormat.Json)]
        byte[] GetVideoBytes(string videofile);


    }
}

This servicecontract consists of two operations (methods) and they are exposed to the WCF Rest programming model through the WebGet attribute and an uri template. The method GetVideo returns a Stream of the video file requested. This will allow the user to play off the video file as a stream and has many benefits. First off, the load time is quick - as soon as enough data is collected (and the video file format supports it), the video can play back. We will be using a video/mp4 format in this demo and it supports Streaming playback. An additional benefit of streaming is that it is lightweight. It has low latency, efficient bandwidth usage and allows the client to spend little memory to fetch data. The downside is that our Stream does not support any other type of playback than uni-directional, starting from the beginning. You cannot fast forward, rewind or go to a location in the video with just a Stream like this.

I have not looked into if we could provide some offset to the Stream and skip to for example a specified time in the video. Instead I have made another method called GetVideoBytes that returns a byte array. A client can then retrieve the entire video as a byte array and then dynamically build up a Blob object and create a Blob url and dynamically set the src of the video. The WCF service implementation looks like this:
 using System.IO;
using System.Web;
using WcfStreamAudioDemo.Common;

namespace WcfStreamAudioDemo.Host
{
    public class VideoService : IVideoServiceContract
    {

        public Stream GetVideo(string videofile)
        {
            return GetVideoStream(videofile, ".mp4");
        }

        public byte[] GetVideoBytes(string videofile)
        {
            Stream videoStream = GetVideoStream(videofile, ".mp4");

            byte[] buffer = new byte[32*1024];
            int count;
            var memoryStream = new MemoryStream();

            while ((count = videoStream.Read(buffer, 0, buffer.Length)) > 0)
            {
                memoryStream.Write(buffer, 0, count);
            }

            return memoryStream.ToArray();
        }

        private Stream GetVideoStream(string videofile, string extension)
        {
            string mediaFolder = HttpContext.Current.Server.MapPath("~/media");
            string videoFileFullPath = Path.Combine(mediaFolder, videofile) + extension;
            return File.OpenRead(videoFileFullPath);
        }

    }
}

Nothing special going on here, we grab hold of a FileStream of the video file using HttpContext.Current.Server.MapPath and Path.Combine to locate it in a media folder (there is no error handling in this simple sample) and then use System.IO.File.OpenRead. The method returning a byte array loops through the Stream using a MemoryStream and then returns a byte array from this stream again. Let's look at the client side code. Sadly, using a responsetype of 'arraybuffer' in a XMLHttpRequest has given bloated data. Instead, I must pass the byte array as raw data to the client and manually mold and bake the data into a working ArrayBuffer that then is used in an Uint8Array as a view for this Buffer and instatiating a Blob in Javascript with a Blob url with URL.createBlobUrl and then setting finally the src attribute of a video control in the page. It is not very elegant and there is a performance penalty, especially for larger video files, it would be problematic. Our sample video is just 4.32 MB and can be handled. Of course, this article is just for educational purposes and shows you what CAN be achieved, optimizing this for production purposes would of course need to get this ArrayBuffer to work more elegant. So here is my manual code to fix up the dynamic loading of a byte array into data that HTML5 Video can use:

    <script type="text/javascript">

        window.onload = init;

       
        var source; //Video buffer 

        function init() {
            loadByteArray('http://wcfaudiodemohost.azurewebsites.net/VideoService.svc/mediabytes/sintel_trailer-480p');
        }

        function loadByteArray(url) {
            
            var request = new XMLHttpRequest();

            request.overrideMimeType('text\/plain; charset=x-user-defined');
            //request.responseType = 'arraybuffer';
            request.open('GET', url, true);
            //request.setRequestHeader("Content-Type", "video/mp4");


            request.onload = function() {
                //console.log(request.response);
                debugger;

                var responseData = request.response;
                if (responseData === undefined || responseData === null)
                    return;

                if (responseData.charAt(0) === '[') {
                    responseData = responseData.substring(1);
                }
                if (responseData !== null &&
                    responseData !== undefined &&
                    responseData.charAt(responseData.length - 1) === ']') {
                    responseData = responseData.slice(0, -1);
                }

                if (responseData === undefined || responseData === null)
                    return;

                var videoByteArray = responseData.split(',');
                source = new ArrayBuffer(videoByteArray.length);

                var videoByteArrayView = new Uint8Array(source);


                for (var i = 0; i < videoByteArray.length; i++) {
                    videoByteArrayView[i] = videoByteArray[i];
                }

                var blob = new Blob([videoByteArrayView], { type: "text/plain;charset=utf-8" });

                var blobUrl = URL.createObjectURL(blob);

                var videoCtrlFedByByteArray = document.getElementById("videoCtrlFedByByteArray");

                videoCtrlFedByByteArray.setAttribute("src", blobUrl);

                saveAs(blob, "SintelTrailer_VideoReceived.mp4");

            } //request.onload 

            request.send();

        }

    </script>

Take note what I am doing here:
  request.overrideMimeType('text\/plain; charset=x-user-defined');
This enforces the byte array to be sent as a raw string. Our video with about 4.37 MB actually is 15.54 MB in Fiddler. Not very efficient. I will look into if this can be fixed up in a future article. Our GUI markup looks like this:
 <div>
        
        <h3>WCF Sample Video streaming example (<strong>System.IO.Stream</strong>)</h3>
        
        <p>Playing an wideo file (.mp4) using a custom WCF IDispatchMessageFormatter with REST (webHttpBinding).</p>
        
        <video controls width="320" height="240">
            <source src="http://wcfaudiodemohost.azurewebsites.net/VideoService.svc/media/sintel_trailer-480p" type="video/mp4" />
            <p>Your browser doesn't support HTML5 video. Here is a <a href="http://wcfaudiodemohost.azurewebsites.net/VideoService.svc/media/sintel_trailer-480p">link to the video</a> instead.</p> 
        </video>
        
        <h3>WCF Sample Video streaming example (<strong>byte[] array</strong>)</h3>
        
        <p>Playing an video file (.mp4) using a custom WCF IDispatchMessageFormatter with REST (webHttpBinding).</p>
        
        <video id="videoCtrlFedByByteArray" controls width="320" height="240">         
            <p>Your browser doesn't support HTML5 video. Here is a <a href="http://wcfaudiodemohost.azurewebsites.net/VideoService.svc/mediabytes/sintel_trailer-480p">link to the video</a> instead.</p> 
        </video>
        
    </div>

As you can see, the Stream method allowed us to set the src directly. The byte[] array instead needed us to do a lot of manual molding of the data. I hope you found this article interesting and helpful.