Loading video from WCF into HTML5 Video

Live demo here!

WCF demo - Loading video using Stream or byte array into HTML5 Video control
This article will look at loading up a video into a HTML5 Video. First off, you can clone the repository I have prepared here:
git clone https://toreaurstad@bitbucket.org/toreaurstad/wcfaudiostreamdemo.git
git fetch && git checkout VideoDemo_04082018

The image below shows our user interface, a simple web page displaying two HTML5 Video controls. This article builds upon code from previous articles. Defining a WCF Service Contract IVideoServiceContract is as following:

using System.IO;
using System.ServiceModel;
using System.ServiceModel.Web;

namespace WcfStreamAudioDemo.Common
{
    
    [ServiceContract]
    public interface IVideoServiceContract
    {

        [OperationContract]
        [ContentType("video/mp4")]
        [WebGet(UriTemplate = "media/{videofile}")]
        Stream GetVideo(string videofile);

        [OperationContract]
        [WebGet(UriTemplate = "mediabytes/{videofile}", ResponseFormat = WebMessageFormat.Json)]
        byte[] GetVideoBytes(string videofile);


    }
}

This servicecontract consists of two operations (methods) and they are exposed to the WCF Rest programming model through the WebGet attribute and an uri template. The method GetVideo returns a Stream of the video file requested. This will allow the user to play off the video file as a stream and has many benefits. First off, the load time is quick - as soon as enough data is collected (and the video file format supports it), the video can play back. We will be using a video/mp4 format in this demo and it supports Streaming playback. An additional benefit of streaming is that it is lightweight. It has low latency, efficient bandwidth usage and allows the client to spend little memory to fetch data. The downside is that our Stream does not support any other type of playback than uni-directional, starting from the beginning. You cannot fast forward, rewind or go to a location in the video with just a Stream like this.

I have not looked into if we could provide some offset to the Stream and skip to for example a specified time in the video. Instead I have made another method called GetVideoBytes that returns a byte array. A client can then retrieve the entire video as a byte array and then dynamically build up a Blob object and create a Blob url and dynamically set the src of the video. The WCF service implementation looks like this:

 using System.IO;
using System.Web;
using WcfStreamAudioDemo.Common;

namespace WcfStreamAudioDemo.Host
{
    public class VideoService : IVideoServiceContract
    {

        public Stream GetVideo(string videofile)
        {
            return GetVideoStream(videofile, ".mp4");
        }

        public byte[] GetVideoBytes(string videofile)
        {
            Stream videoStream = GetVideoStream(videofile, ".mp4");

            byte[] buffer = new byte[32*1024];
            int count;
            var memoryStream = new MemoryStream();

            while ((count = videoStream.Read(buffer, 0, buffer.Length)) > 0)
            {
                memoryStream.Write(buffer, 0, count);
            }

            return memoryStream.ToArray();
        }

        private Stream GetVideoStream(string videofile, string extension)
        {
            string mediaFolder = HttpContext.Current.Server.MapPath("~/media");
            string videoFileFullPath = Path.Combine(mediaFolder, videofile) + extension;
            return File.OpenRead(videoFileFullPath);
        }

    }
}

Nothing special going on here, we grab hold of a FileStream of the video file using HttpContext.Current.Server.MapPath and Path.Combine to locate it in a media folder (there is no error handling in this simple sample) and then use System.IO.File.OpenRead. The method returning a byte array loops through the Stream using a MemoryStream and then returns a byte array from this stream again. Let's look at the client side code. Sadly, using a responsetype of 'arraybuffer' in a XMLHttpRequest has given bloated data. Instead, I must pass the byte array as raw data to the client and manually mold and bake the data into a working ArrayBuffer that then is used in an Uint8Array as a view for this Buffer and instatiating a Blob in Javascript with a Blob url with URL.createBlobUrl and then setting finally the src attribute of a video control in the page. It is not very elegant and there is a performance penalty, especially for larger video files, it would be problematic. Our sample video is just 4.32 MB and can be handled. Of course, this article is just for educational purposes and shows you what CAN be achieved, optimizing this for production purposes would of course need to get this ArrayBuffer to work more elegant. So here is my manual code to fix up the dynamic loading of a byte array into data that HTML5 Video can use:

    <script type="text/javascript">

        window.onload = init;

       
        var source; //Video buffer 

        function init() {
            loadByteArray('http://wcfaudiodemohost.azurewebsites.net/VideoService.svc/mediabytes/sintel_trailer-480p');
        }

        function loadByteArray(url) {
            
            var request = new XMLHttpRequest();

            request.overrideMimeType('text\/plain; charset=x-user-defined');
            //request.responseType = 'arraybuffer';
            request.open('GET', url, true);
            //request.setRequestHeader("Content-Type", "video/mp4");


            request.onload = function() {
                //console.log(request.response);
                debugger;

                var responseData = request.response;
                if (responseData === undefined || responseData === null)
                    return;

                if (responseData.charAt(0) === '[') {
                    responseData = responseData.substring(1);
                }
                if (responseData !== null &&
                    responseData !== undefined &&
                    responseData.charAt(responseData.length - 1) === ']') {
                    responseData = responseData.slice(0, -1);
                }

                if (responseData === undefined || responseData === null)
                    return;

                var videoByteArray = responseData.split(',');
                source = new ArrayBuffer(videoByteArray.length);

                var videoByteArrayView = new Uint8Array(source);


                for (var i = 0; i < videoByteArray.length; i++) {
                    videoByteArrayView[i] = videoByteArray[i];
                }

                var blob = new Blob([videoByteArrayView], { type: "text/plain;charset=utf-8" });

                var blobUrl = URL.createObjectURL(blob);

                var videoCtrlFedByByteArray = document.getElementById("videoCtrlFedByByteArray");

                videoCtrlFedByByteArray.setAttribute("src", blobUrl);

                saveAs(blob, "SintelTrailer_VideoReceived.mp4");

            } //request.onload 

            request.send();

        }

    </script>

Take note what I am doing here:

  request.overrideMimeType('text\/plain; charset=x-user-defined');

This enforces the byte array to be sent as a raw string. Our video with about 4.37 MB actually is 15.54 MB in Fiddler. Not very efficient. I will look into if this can be fixed up in a future article. Our GUI markup looks like this:

 <div>
        
        <h3>WCF Sample Video streaming example (<strong>System.IO.Stream</strong>)</h3>
        
        <p>Playing an wideo file (.mp4) using a custom WCF IDispatchMessageFormatter with REST (webHttpBinding).</p>
        
        <video controls width="320" height="240">
            <source src="http://wcfaudiodemohost.azurewebsites.net/VideoService.svc/media/sintel_trailer-480p" type="video/mp4" />
            <p>Your browser doesn't support HTML5 video. Here is a <a href="http://wcfaudiodemohost.azurewebsites.net/VideoService.svc/media/sintel_trailer-480p">link to the video</a> instead.</p> 
        </video>
        
        <h3>WCF Sample Video streaming example (<strong>byte[] array</strong>)</h3>
        
        <p>Playing an video file (.mp4) using a custom WCF IDispatchMessageFormatter with REST (webHttpBinding).</p>
        
        <video id="videoCtrlFedByByteArray" controls width="320" height="240">         
            <p>Your browser doesn't support HTML5 video. Here is a <a href="http://wcfaudiodemohost.azurewebsites.net/VideoService.svc/mediabytes/sintel_trailer-480p">link to the video</a> instead.</p> 
        </video>
        
    </div>

As you can see, the Stream method allowed us to set the src directly. The byte[] array instead needed us to do a lot of manual molding of the data. I hope you found this article interesting and helpful.

Loading HTML5 Audio from byte array in WCF

Update:

Live demo!

Wcf Audio Demo - HTML 5 and Byte array!

The previous article presented a solution where the user can retrieve data from a WCF operation that returns a Stream. This approach is great, since the user can quickly load data and progressively retrieve it. This makes an efficient use of bandwidth and avoids latency. However, there is a downside to this approach. It is not possible to skip parts of the loaded Stream, so the user can only start from the beginning of the Stream. The article presented loading a Waveform Audio Format file (audio/wav). Another approach is instead to let the user play, rewind and skip to parts of the data, be it a video or audio file for example. This article will present a solution of this and also solve it in memory by returning a byte array from WCF and load it into the WebAudio AudioContext of a webpage and then set the src of a HTML5 Audio element. First off, you can clone my repository which I have prepared from Bitbucket and switch to the branch supporting a byte array.

 git clone https://toreaurstad@bitbucket.org/toreaurstad/wcfaudiostreamdemo.git
 git checkout ByteArrayWise_03082018

Let's first return a byte array to the client in a WCF service contract. We will return a byte array of a sample ogg/vorbis (mime: audio/ogg) file. The WCF Service contract now got a new method GetAudioBytes :

    [ServiceContract]
    public interface IAudioServiceContract
    {

        [OperationContract]
        [ContentType("audio/ogg")]
        [WebGet(UriTemplate = "media/{trackName}")]
        Stream GetAudio(string trackName);

        [OperationContract]
        [WebGet(UriTemplate = "mediabytes/{trackName}", ResponseFormat = WebMessageFormat.Json)]
        byte[] GetAudioBytes(string trackName);

    }

Next off, we implement the method GetAudioBytes :

        private static Stream GetStream(string trackName, string fileExtension)
        {
            var mediaDirectory = HttpContext.Current.Server.MapPath("~/media/");
            string track = string.Format("{0}.{1}", trackName, fileExtension);
            string filePath = Path.Combine(mediaDirectory, track);
            return File.OpenRead(filePath);
        }

        public byte[] GetAudioBytes(string trackName)
        {
            Stream track = GetStream(trackName, "ogg");

            long streamLengh = track.Length;
            byte[] buffer = new byte[32*1024];
            int count;

            var memoryStream = new MemoryStream();
            while ((count = track.Read(buffer, 0, buffer.Length)) > 0)
            {
                memoryStream.Write(buffer, 0, count);
            }

            return memoryStream.ToArray();
        }

The reading of the Stream into a byte array is done via a memory stream with a buffer of size 32 kB. I wrote the method above and checked with one of Jon Skeet's advice and it was identical :) e We now have got a byte array, then the client must get this byte array. Since I decided to create a web client, it is necessary to make use of a client library. I will use HTML 5 audio and WebAudio API for this. First off, we make an XMLHttpRequest to fetch the data, we set the responseType of the request to 'arraybuffer'. I encountered trouble in Javascript sending the byte array from WCF and ended up to force Js to send the data as plain text raw format. Then I had to manually build an arraybuffer by using an Uint8Array as a view to do the actual insertion. There is a lot of manual work done here to ultimately populate the ArrayBuffer with exact same bytes as returned from the WCF service. As I said, by first trying a manual retrieval, I ended up with bloated data that I had a to bake into shape by forcing a plain retrieval of data and then molding the data into an ArrayBuffer. Not very elegant, but it works.

    
    <script src="Scripts/FileSaver.js" type="text/javascript"></script>
    
    <script type="text/javascript">

        window.onload = init;

        var context; //AudioContext
        var source; //Audio buffer 

        function init() {
            if (!window.AudioContext) {
                alert("Your browser does not support any Audio Context and cannot play back this audio");
                return;
            }
            window.AudioContext = window.AudioContext || window.webkitAudioContext;

            context =  typeof AudioContext !== 'undefined' ? new AudioContext() : new webkitAudioContext();;

            loadByteArray('http://myserver/WcfStreamAudioDemo.Host/AudioService.svc/mediabytes/ACDC_-_Back_In_Black-sample');
        }

        function loadByteArray(url) {
            source = context.createBufferSource();
            var request = new XMLHttpRequest();
           
            request.overrideMimeType('text\/plain; charset=x-user-defined');
            //request.responseType = 'arraybuffer';
            request.open('GET', url, true);
            //request.setRequestHeader("Content-Type", "audio/ogg");


            request.onload = function() {
                //console.log(request.response);
                debugger;

                var audioByteArray = request.response.replace('[', '').replace(']','').split(',');
                source = new ArrayBuffer(audioByteArray.length);

                var audioByteArrayView = new Uint8Array(source);


                for (var i = 0; i < audioByteArray.length; i++) {
                    audioByteArrayView[i] = audioByteArray[i];
                }

                var blob = new Blob([audioByteArrayView], { type: "text/plain;charset=utf-8" });

                var blobUrl = URL.createObjectURL(blob);

                var audioCtrlFedByByteArray = document.getElementById("audioCtrlFedByByteArray");

                audioCtrlFedByByteArray.setAttribute("src", blobUrl);



                saveAs(blob, "ExampleReceived.ogg");
             
                context.decodeAudioData(source,
                    function (buffer) {
                        //Play off the audio file automatically
                        var innersource = context.createBufferSource();
                        innersource.buffer = buffer;
                        innersource.connect(context.destination);

                        innersource.start(0);

                      
                        //Hook up the HTML 5 audio control using MediaElementSource connecting to the AudioContext
                        var audioCtrlFedByByteArray = document.getElementById("audioCtrlFedByByteArray");
  
                        var mediaSource = context.createMediaElementSource(audioCtrlFedByByteArray);
                        mediaSource.connect(context.destination);
                        audioCtrlFedByByteArray.play();

                    },
                    function(e) {
                        console.log("Error with decoding audio data" + e.err);
                    }
            );

        }

            request.send();

        }

    </script>

There are three techniques presented into the client-side script. First off I show how you automtically can play downloaded audio into a web page! I use the AudioContext first and decodeAudioData by creating a bufferSource and connect that source to the AudioContext's destination (i.e. speakers). Then i choose start(0) of the bufferSource to automatically playback! Also demonstrated in the Javascript above is how I grab hold of the audio control and then setting the src attribute to a blobUrl created by URL.createObjectUrl feeding it a Blob (Binary large object) of the created Uint8Array populated by the raw data. This will set dynamically the source of the HTML 5 Audio media element on the page and let the user finally play the audio file and rewind or skip to different parts! It is much more user friendly, but the user will have to download the entire file first and the manual process by molding the retrieved raw data into a working ArrayBuffer with correct data will be slow when dealing with large files. The web page of the client defines the audio controls like this:

rm id="HtmlForm" runat="server">
    <div>
        
        <h3>WCF Sample Audio streaming example (<strong>System.IO.Stream</strong>)</h3>
        
        <p>Playing an audio file (.wav) using a custom WCF IDispatchMessageFormatter with REST (webHttpBinding).</p>
        
        <audio controls>
            <source src="http://myserver/WcfStreamAudioDemo.Host/AudioService.svc/media/Example" type="audio/ogg" />
            <p>Your browser doesn't support HTML5 audio. Here is a <a href="http://myserver/WcfStreamAudioDemo.Host/AudioService.svc/media/Example">link to the audio</a> instead.</p> 
        </audio>
        
        <h3>WCF Sample Audio streaming example (<strong>byte[] array</strong>)</h3>
        
        <p>Playing an audio file (.wav) using a custom WCF IDispatchMessageFormatter with REST (webHttpBinding).</p>
        
        <audio id="audioCtrlFedByByteArray" controls>         
            <p>Your browser doesn't support HTML5 audio. Here is a <a href="http://myserver/WcfStreamAudioDemo.Host/AudioService.svc/mediabytes/ACDC_-_Back_In_Black-sample">link to the audio</a> instead.</p> 
        </audio>
        
    </div>
    

Make note that my sample page also includes the library FileSaver to download the ogg sample file. This library is quite handy and is available in Nuget (package id: filesaver). We finally have a way to load data from WCF into a HTML 5 Audio control in memory! This means we could for example have audio tracks saved into a database and fetch them into memory and return them to our clients. An alternative is of course to just return a FileStream and have a server backend written in ASP.NET MVC Core, but then you will also be missing a lot of the nice features of WCF! All in all, it is always nice to know alternatives before sticking to the mainstream solutions. I hope you found this article interesting.