Programatically compress data contracts saving bandwidth between clients and WCF

This article will explain how to achieve compression when transferring data contracts between a WCF service and a client. Many developers using WCF have used the tool Fiddler or similar to inspect the network traffic. The default transmission of data between services and clients are data contracts that are serialized using the SOAP XML protocol. Note that this will usually send data uncompressed as XML of course. This is not an issue for smaller data contracts, but when you start transmitting much data, data contracts can soon grow into MegaBytes (MB) of data and the TTF (Time-To-Transfer) gets noticably, esecially on low-bandwidth devices such as smart phones! It is possible to configure compression in IIS, but more control can be achieved with a mini-framework of mine I have developed and I will present next. First we need some code to be able to compress data. The following class, GzipByteArrayCompressionutility, uses the MemoryStream and BufferedStream in the System.IO namespace and the GZipStream class in the System.IO.Compression namespace:

GzipByteArrayCompressionUtility

using System;
using System.IO;
using System.IO.Compression;

namespace SomeAcme.SomeProduct.Common.Compression
{

    public static class GzipByteArrayCompressionUtility
    {

        private static readonly int bufferSize = 64 * 1024; //64kB

        public static byte[] Compress(byte[] inputData)
        {
            if (inputData == null)
                throw new ArgumentNullException("inputData must be non-null");

            using (var compressIntoMs = new MemoryStream())
            {
                using (var gzs = new BufferedStream(new GZipStream(compressIntoMs,
                 CompressionMode.Compress), bufferSize))
                {
                    gzs.Write(inputData, 0, inputData.Length);
                }
                return compressIntoMs.ToArray();
            }
        }

        public static byte[] Decompress(byte[] inputData)
        {
            if (inputData == null)
                throw new ArgumentNullException("inputData must be non-null");

            using (var compressedMs = new MemoryStream(inputData))
            {
                using (var decompressedMs = new MemoryStream())
                {
                    using (var gzs = new BufferedStream(new GZipStream(compressedMs,
                     CompressionMode.Decompress), bufferSize))
                    {
                        gzs.CopyTo(decompressedMs);
                    }
                    return decompressedMs.ToArray();
                }
            }
        }

        //private static void Pump(Stream input, Stream output)
        //{
        //    byte[] bytes = new byte[4096];
        //    int n;
        //    while ((n = input.Read(bytes, 0, bytes.Length)) != 0)
        //    {
        //        output.Write(bytes, 0, n); 
        //    }
        //}

    }


}

Further, it is necessary to have some utility methods to handle data contracts and performing both the compression and decompression. I have also added some handy methods in this class for cloning data contracts.

DataContractUtility

using System;
using System.Collections.Generic;
using System.Configuration;
using System.IO;
using System.Runtime.Serialization;
using System.Text;
using System.Xml;
using System.Xml.Serialization;


namespace SomeAcme.SomeProduct.Common
{

    /// <summary>
    /// Utility methods for serializing, deserializing and cloning data contracts
    /// </summary>
    /// <remarks>The serialization, deserialization and cloning here is 
    /// Limited to data contracts, as DataContractSerializer is being used for these operations</remarks>
    public static class DataContractUtility
    {

        public static byte[] ToByteArray<T>(T dataContract) where T : class
        {
            if (dataContract == null)
            {
                throw new ArgumentNullException();
            }

            using (MemoryStream memoryStream = new MemoryStream())
            {
                var serializer = new DataContractSerializer(typeof(T));
                serializer.WriteObject(memoryStream, dataContract);
                memoryStream.Position = 0;
                return memoryStream.ToArray();                
            }
        }

        public static List<TResult> GetDataContractsFromDataContractContainer<TContainer, TResult>(TContainer container, CompressionSetupDataContract setup) 
            where TContainer : class, IDataContractContainer<TResult>  
            where TResult : class
        {
            if (container == null)
                return null;
            if (!container.IsBinary || !IsUseNetworkCompression(setup))
                return container.SerialPayload;
            else
            {
                byte[] unzippedData = GzipByteArrayCompressionUtility.Decompress(container.BinaryPayload);
                var dataContracts = DataContractUtility.DeserializeFromByteArray<List<TResult>>(unzippedData);
                return dataContracts;
            }
        }

        private static bool IsUseNetworkCompression(CompressionSetupDataContract setup)
        {
            if (setup != null)
                return setup.UseNetworkCompression;

            if (ConfigurationManager.AppSettings[Constants.UseNetworkCompression] == null)
                return false; 

            bool isUseNetworkCompression = false;
            if (!bool.TryParse(ConfigurationManager.AppSettings[Constants.UseNetworkCompression], out isUseNetworkCompression))
                return false;
            else
                return isUseNetworkCompression;            
        }

        private static int GetNetworkCompressionItemTreshold(CompressionSetupDataContract setup)
        {
            if (setup != null)
                return setup.NetworkCompressionItemTreshold; 

            int standardNetworkCompressionItemTreshold = 100;
            if (ConfigurationManager.AppSettings[Constants.NetworkCompressionItemTreshold] == null)
                return standardNetworkCompressionItemTreshold;
            int networkCompressionItemTreshold = standardNetworkCompressionItemTreshold;
            if (!int.TryParse(ConfigurationManager.AppSettings[Constants.NetworkCompressionItemTreshold], out networkCompressionItemTreshold))
                return standardNetworkCompressionItemTreshold;
            else
                return networkCompressionItemTreshold;
        }



        public static TContainer GetContainerInstanceAfterResult<TContainer, TResult>(List<TResult> result, CompressionSetupDataContract setup) 
            where TContainer : class, IDataContractContainer<TResult>, new()
            where TResult : class
        {
            TContainer container = new TContainer();

            if (IsUseNetworkCompression(setup) && (result != null && result.Count >= Math.Min(container.BinaryTreshold, GetNetworkCompressionItemTreshold(setup))))
            {
                if (container.IsGzipped)
                    container.BinaryPayload = GzipByteArrayCompressionUtility.Compress(DataContractUtility.ToByteArray(result));
                else 
                    container.BinaryPayload = DataContractUtility.ToByteArray(result); 
                
                container.IsBinary = true;
            }
            else
            {
                container.SerialPayload = result;
            }

            return container; 
        }

        public static string SerializeObject<T>(T dataContract) where T : class
        {
            using (var memoryStream = new MemoryStream())
            {
                using (var streamReader = new StreamReader(memoryStream))
                {
                    var serializer = new DataContractSerializer(typeof(T));
                    serializer.WriteObject(memoryStream, dataContract);
                    memoryStream.Position = 0;
                    return streamReader.ReadToEnd();
                }
            }
        }

        public static T DeserializeObject<T>(string serializedContent) where T : class
        {
            return DeserializeObject<T>(serializedContent, Encoding.UTF8);
        }

        public static T DeserializeObject<T>(string serializedContent, Encoding encoding) where T : class
        {
            T result = null;
            using (Stream memoryStream = new MemoryStream())
            {
                var serializer = new DataContractSerializer(typeof(T));
                byte[] data = encoding.GetBytes(serializedContent);
                memoryStream.Write(data, 0, data.Length);
                memoryStream.Position = 0;
                result = (T)serializer.ReadObject(memoryStream);
            }
            return result;
        }

        public static T DeserializeFromByteArray<T>(byte[] data) where T : class
        {
            T result = null;
            using (Stream memoryStream = new MemoryStream())
            {
                var serializer = new DataContractSerializer(typeof(T));
                memoryStream.Write(data, 0, data.Length);
                memoryStream.Position = 0;
                result = (T)serializer.ReadObject(memoryStream);
            }
            return result;
        }

        public static T CloneObject<T>(T dataContract) where T : class
        {
            T result = null;
            var serializedContent = SerializeObject<T>(dataContract);
            result = DeserializeObject<T>(serializedContent);
            return result;
        }

    }

}

Note that I use two app settings here to control the config of compression in web.config:

    <add key="UseNetworkCompression" value="true" />
    <add key="NetworkCompressionItemTreshold" value="100" />

The appsetting UseNetworkCompression turns on and off the compression. If compression is turned off, we can switch to default SOAP XML serialization. But if it is turned on, the data will be packet into a compressed byte array. The appsetting NetworkCompressionItemTreshold is the minimum number of data contract items that will trigger compression. Here we use two constants also:

        public const string UseNetworkCompression = "UseNetworkCompression";
        public const string NetworkCompressionItemTreshold = "NetworkCompressionItemTreshold";

That was some of code to handle the compression and decompression of data contracts, next I show sample code how to use this. First we need to create a "container data class" for a demo of this mini framework. Each class that will be a "container data contract" will implement the interface IDataContractContainer:

using System.Collections.Generic;



    public interface IDataContractContainer<TResult> where TResult : class
    {

        /// 

        /// Set to true if the binary payload is to be used
        /// 
        bool IsBinary { get; set; }

        /// 

        /// If not is binary, the serial payload contains the data (SOAP XML based data contracts)
        /// 
        List<TResult> SerialPayload { get; set; }

        /// 

        /// Byte array which is the binary payload. Usually the byte array is also Gzipped.
        /// 
        byte[] BinaryPayload { get; set; }

        /// 

        /// The limit when the use of binary payload should be honored.
        /// 
        int BinaryTreshold { get; set; }

        /// 

        /// If true, the binary payload is Gzipped (compressed)
        /// 
        bool IsGzipped { get; set; }

    }

The following class implements the IDataContractContainer interface:

using System.Collections.Generic;
using System.Runtime.Serialization; 


   
    [DataContract(Namespace=Constants.DataContractNamespace20091001)]
    public class OperationItemsContainerDataContract : IDataContractContainer<OperationItemDataContract>
    {

        public OperationItemsContainerDataContract()
        {
            BinaryTreshold = 100;
            IsGzipped = true;
        }

        [DataMember(Order = 1)]
        public byte[] BinaryPayload { get; set; }

        [DataMember(Order = 2)]
        public List<OperationItemDataContract> SerialPayload { get; set; }

        [DataMember(Order = 3)]
        public bool IsBinary { get; set; }

        [DataMember(Order = 4)]
        public int BinaryTreshold { get; set; }

        [DataMember(Order = 5)]
        public bool IsGzipped { get; set; }

    }

Next up is some example code how to use this code:

        public OperationItemsContainerDataContract GetOperationItems(OperationsRequestDataContract request)
        {

..
            var operations = SomeManager.GetSomeData(rquest);
..


            OperationItemsContainerDataContract result =
                DataContractUtility.GetContainerInstanceAfterResult<OperationItemsContainerDataContract, OperationItemDataContract>(operations, null);
            
            return result;
        }

The code above includes some production code, but what matters here is the call to DataContractUtility.GetContainerInstanceAfterResult. This will compress the data contracts into a byte array, if the container instance defines this. Note that the compression will actually pack the data contract items into a byte array and apply GZip compression if the container class set this up. See the constructor of the OperationItemsContainer class, where we set GZip compression to be used if we have activated compression in web.config and the result set is above the limit set up in web.config On the client side, it is necessary retrieve the data. On the server side we have used the GetContainerInstanceAfterResult method, we will now use the GetDataContractsFromDataContractContainer method.

class OperationItemProvider:

  private void LoadDailyOperationsByCurrentTheater(Collection operationItems)
        {
           
            try
            {
                var container = SomeAgent.GetSomeOperationItems(request);


                var operations = DataContractUtility.GetDataContractsFromDataContractContainer<OperationItemsContainerDataContract, OperationItemDataContract>(container, Context.CompressionSetup);

        
            }
            catch (SomeAcmeClientException ex)
            {
                DispatcherUtil.InvokeAction(() => EventAggregator.GetEvent<ErrorMessageEvent>().Publish(new ErrorMessageEventArg { ErrorMessage = ex.Message }));
            }

..
        
        }

Note that the code above includes some production code. The important part is the use of the method GetDataContractsFromDataContractContainer. The object Context.CompressionSetup contains our web.config and is a simple retrieval of the configured settings on our serverside (the serverside method uses the ConfigurationManager. Here is the WCF server method:

        public CompressionSetupDataContract GetCompressionSetup()
        {
            bool standardUseNetworkCompression = false;
            int standardNetworkCompressionItemTreshold = 100;       
           
            var setup = new CompressionSetupDataContract
            {
                UseNetworkCompression = standardUseNetworkCompression, 
                NetworkCompressionItemTreshold = standardNetworkCompressionItemTreshold
            };
            if (ConfigurationManager.AppSettings[Constants.UseNetworkCompression] != null)
            {
                bool useNetworkCompression = standardUseNetworkCompression;
                setup.UseNetworkCompression = bool.TryParse(ConfigurationManager.AppSettings[SomeAcme.SomeProduct.Common.Constants.UseNetworkCompression], out useNetworkCompression) ?
                    useNetworkCompression : false;
            }

            if (ConfigurationManager.AppSettings[Constants.NetworkCompressionItemTreshold] != null)
            {
                int networkCompressionItemTreshold = standardNetworkCompressionItemTreshold;
                setup.NetworkCompressionItemTreshold = 
                    int.TryParse(ConfigurationManager.AppSettings[SomeAcme.SomeProduct.Common.Constants.NetworkCompressionItemTreshold], out networkCompressionItemTreshold) ?
                    networkCompressionItemTreshold : standardNetworkCompressionItemTreshold;
            }          

            return setup; 
        }

We also need to control the use of compression to be able to run integration tests on this:

        [ServiceLog]
        [AuthorizedRole(SomeAcmeRoles.Administrator)]
        public bool SetUseNetworkCompression(CompressionSetupDataContract compressionSetup)
        {
            if (compressionSetup == null)
                throw new ArgumentNullException(GetName.Of(() =< compressionSetup));
            try
            {
                //ADDITIONAL SECURITY CHECKS OMITTED FROM PUBLIC DISPLAY.

                System.Configuration.Configuration webConfigApp = WebConfigurationManager.OpenWebConfiguration("~");

                webConfigApp.AppSettings.Settings[Common.Constants.UseNetworkCompression].Value = compressionSetup.UseNetworkCompression
                    ? "true"
                    : "false";

                webConfigApp.AppSettings.Settings[Common.Constants.NetworkCompressionItemTreshold].Value = compressionSetup.NetworkCompressionItemTreshold.ToString(); 

                webConfigApp.Save();
                return true;
            }
            catch (Exception err)
            {
                InterfaceBinding.GetInstance().WriteError(err.Message);
                return false;
            }
        }

Okay, so no we have the necessary code to test out the compression. Of course to apply this mini-framework to your codebase, you will need to add at least the two app settings above and the two utility classes presented, plus the IDataContractContainer, an implementation of this and use the two methods on the service side and the client side of DataContractUtility noted - To note again:

On the client side, it is necessary retrieve the data. On the server side we have used the GetContainerInstanceAfterResult method, we will now use the GetDataContractsFromDataContractContainer method.

Here is an integration test for testing out our mini framework:

        [Test]
        [Category(TestCategories.IntegrationTest)]
        public void GetOperationItemsWithCompressionEnabledDoesNotReturnEmpty()
        {
            var compressionSetup = new CompressionSetupDataContract
            {
                UseNetworkCompression = true,
                NetworkCompressionItemTreshold = 1
            };

            SystemServiceAgent.SetUseNetworkCompression(compressionSetup);

            //Arrange 
            var operationsRequest = new OperationsRequestDataContract()
            {
                .. // some init
            };

            //Act 
            OperationItemsContainerDataContract operationsRetrieved = ConcreteServiceAgent.GetSomeOperationItems(operationsRequest);

            //Assert 
            Assert.IsNotNull(operationsRetrieved);

            var operationItems =
                DataContractUtility.GetDataContractsFromDataContractContainer<OperationItemsContainerDataContract, OperationItemDataContract>(
                    operationsRetrieved, compressionSetup);

            Assert.IsNotNull(operationItems);
            CollectionAssert.IsNotEmpty(operationItems);

            compressionSetup.UseNetworkCompression = false;
            compressionSetup.NetworkCompressionItemTreshold = 100;
            SystemServiceAgent.SetUseNetworkCompression(compressionSetup);
        }

Note that the compression will mean that more CPU is used on the service side. This can in many cases mean that while saving bandwidth, you spend more CPU resources on an already busy application server running the WCF services. At the same time, more and more clients of WCF services are on low-bandwidth devices, such as smart phones. I have run some tests and retrieved about 1000 items of relateively large data contracts and witnessed a saving from 5-6 MegaBytes (MB) of data being transferred down to 300 kB, a saving of about 20x! WCF serialization using SOAP XML results often in gigantic amounts of data being transferred. If you are on a Gigabit Ethernet network, it might not be noticably on a developer computer, but if you are creating a system with many simultaneous users, network bandwidth usage is starting to get really important, even on high-bandwidth clients! At the end, I must note that the code presented here is very generic. It can be used not only for sending data between WCF services to clients, but a requirement is that the data you are about to send is data contract items (classes and properties using the DataContract and DataMember attributes.

Share this article on LinkedIn.