Thursday, 30 June 2016

Creating TPL Dataflow meshes to construct pipelines of computations

The TPL DataFlow Library allows the creation of simple and more complex data meshes that propagate data computations and exceptions using the Nuget package Microsoft.Tpl.DataFlow Let's look at how we can create a compound mesh to do three calculations that is considered as a single mesh. These simple examples appear to give simple computations as these a huge overhead in complexity. Of course, you would use Microsoft.Tpl.DataFlow for more complex scenarios, the simple example is just used for clarity. Consider the following code: First off, make sure you add a reference to Microsoft.Tpl.Dataflow, since TPL Dataflow is not part of the base class Library BCL in .NET. In the Nuget Package Explorer commandline in VS:
Install-Package Microsoft.Tpl.DataFlow

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using System.Threading.Tasks.Dataflow;

namespace DataFlowDemo
{

    class Program
    {

        static void Main(string[] args)
        {
            //TplDataDemo();
            SecondTplDataDemo();
            Console.WriteLine("Press any key to continue ..");
            Console.ReadKey();
        }

        private static async void SecondTplDataDemo()
        {
            int[] nums = { 1, 13, 26, 14, 29, 15 };
            Console.WriteLine("Input numbers: ");
            foreach (var n in nums)
                Console.WriteLine(n);
            IPropagatorBlock<int, int> compountBlock = GetPropagatorBlock();
            Console.WriteLine("Pipeline: " + "x = (x * 2) => (x + 2) => (x / 2)");
            foreach (var num in nums)
            {
                compountBlock.Post(num);
            }
            try
            {

                while (true)
                {
                    try
                    {
                        Task<int> f = compountBlock.ReceiveAsync(TimeSpan.FromSeconds(1));
                        await f;
                        await Task.Delay(1000);
                        Console.WriteLine(f.Result);
                    }
                    catch (TimeoutException err)
                    {
                        //Console.WriteLine(err.Message);
                        break;
                    }
                    catch (Exception err)
                    {
                        //Console.WriteLine(err.Message);
                        throw err;
                    }
                }

            }
            catch (Exception err)
            {
                Console.WriteLine(err.Message);
            }
        }

        private static IPropagatorBlock<int, int> GetPropagatorBlock()
        {
            var multiplyBlock = new TransformBlock<int, int>(item => item * 2);
            var addBlock = new TransformBlock<int, int>(item => item + 2);
            var divideBlock = new TransformBlock<int, int>(item => item / 2);

            var flowCompletion = new DataflowLinkOptions { PropagateCompletion = true };
            multiplyBlock.LinkTo(addBlock, flowCompletion);
            addBlock.LinkTo(divideBlock, flowCompletion);

            return DataflowBlock.Encapsulate(multiplyBlock, divideBlock);
        }
  }

We build up the steps of the computation pipeline as a TransformBlock. The multiplyblock is linked to the addBlock and the divideBlock is then linked to the addBlock. We got a pipeline like this: multiplyBlock-addBlock-divideBlock. Each computation will then be: y = (x * 2) => z = y + 2 => w = z / 2. We also use the Encapsulate method to glue together the start step and the end step. We then get the following output:
Input numbers:
1
13
26
14
29
15
Pipeline: x = (x * 2) => (x + 2) => (x / 2)
2
14
27
15
30
16
Press any key to continue ..
Test out TPL Dataflow sample above (VS 2015 solution here: VS Solution With sample code above (.zip)

Wrapping Asynchronous Programming Model (APM) to Task-based Asynchronous Pattern (TAP)

Let's look at how we can wrap classic Begin and End methods used in APM to the newer Task-based Asynchronous Pattern (TAP). Many methods of older framework Versions of .NET support such APM methods and we want to wrap or adapt them to support TAP and async await. Example code:

using System;
using System.IO;
using System.Net;
using System.Text;
using System.Threading.Tasks;

namespace ApmToTap
{
    class Program
    {

        static void Main(string[] args)
        {
            DownloadDemo();

            Console.WriteLine();
            Console.ReadKey();
        }

        private static async void DownloadDemo()
        {
            WebRequest wr = WebRequest.Create("https://t.co/UrkiLgN1BC");
            try
            {
                var response = await wr.GetResponseFromAsync();
                using (Stream stream = response.GetResponseStream())
                {
                    StreamReader reader = new StreamReader(stream, Encoding.UTF8);
                    Console.WriteLine(reader.ReadToEnd());
                }
            }
            catch (Exception err)
            {
                Console.WriteLine(err.Message);
            }
        }
    }

    public static class WebRequestExtensions
    {

        public static Task<WebResponse> GetResponseFromAsync(this WebRequest request)
        {
            return Task<WebResponse>.Factory.FromAsync(request.BeginGetResponse,
                request.EndGetResponse, null);
        }

    }

}

We use the Task<T>Factory.FromAsync method and provide the delegates for the Begin and End methods used in APM. We then provide just null as the AsyncState parameter, as this is not needed. We then can await the Task we create here and get the functionality Task provides such as information of how the asynchronous operation went, exceptions and so on. And of course we can also get the result we usually retrieve in the End method using APM. So there you have it. To use TAP With APM methods, you can use the Task<T>FromAsync method.

Wednesday, 29 June 2016

High performance Producer-Consumer scenario using Nito.AsyncEx

The Nuget Library "Nito.AsyncEx" contains powerful collections that makes it possible to create asynchronous collections that support Producer-Consumer scenarios. First off, let us install the Nuget package:

Install-Package Nito.AsyncEx


using Nito.AsyncEx;
using System;
using System.Collections.Concurrent;
using System.Threading.Tasks;

namespace AsyncCollectionDemo
{
    class Program
    {

        private static readonly AsyncCollection<int> _asyncStack = new AsyncCollection<int>(new ConcurrentStack<int>(), maxCount: 1);


        static void Main(string[] args)
        {
            ProducerConsumerDemo();
        }

        private static async void ConsumerThread()
        {
            await Task.Run(async () =>
             {
                 //Consumer code 
                 while (await _asyncStack.OutputAvailableAsync())
                 {
                     Console.WriteLine(await _asyncStack.TakeAsync());
                     //Thread.Sleep(1000);
                 }
             });
        }

        private static async void ProducerConsumerDemo()
        {
            ConsumerThread();
            //Producer code 
            await _asyncStack.AddAsync(7);
            await _asyncStack.AddAsync(13);
            await _asyncStack.AddAsync(19);
            _asyncStack.CompleteAdding();


            Console.WriteLine("Press any key to continue ..");
            Console.ReadKey(); 
        }

    }
}

We can use the AsyncCollection to create one or multiple producer threads and then return the results to the consumer threads. The benefit of this Collection compared to the BlockingCollection in BCL is that since it supports async, the consumer can be an UI thread for example. So you can have code that produces results and delivers them back to the user Interface. You can ofcourse skip using the ConcurrentStack if you want a FIFO ordering instead of the stack's LIFO ordering. The creator of Nito.AsyncEx is created by Stephen Cleary, which also is the author of the good book "Concurrency in C# Cookboox" which is an O'Reilly book.

Producer-Consumer scenario in .NET using BlockingCollection

The BCL contains helpful classes to build your producer-consumer scenarios. We want to have a thread or several that produces data and one or several threads that consume that data. In this article I will show a simple single producer, single consumer scenario. BlockingCollection contains the necessary logic to support this. Consider the following code:


using System;
using System.Collections.Concurrent;
using System.Threading;
using System.Threading.Tasks;

namespace BlockingCollection
{
    class Program
    {

        private static readonly BlockingCollection<int> _blockingQueue = new BlockingCollection<int>(boundedCapacity: 1); 


        static void Main(string[] args)
        {
            Task.Run(() =>
            {
                foreach (var item in _blockingQueue.GetConsumingEnumerable())
                {
                    Console.WriteLine("Hey I got this item: " + item + "!");
                    Thread.Sleep(1000);
                }

            });

            int[] nums = { 1, 3, 8, 11, 94, 28 };
            foreach (var n in nums)
                _blockingQueue.Add(n);
            _blockingQueue.CompleteAdding();

            Console.WriteLine("Hit the any key!");
            Console.ReadKey(); 
        }

    }
}


We create a blocking collection with a bound capacity of one item. This will only accept a single item at a time. This means that our second call to the .Add() method will actually block. We also use the .CompleteAdding() method to signal that we are finished filling up values. It is also important to start up our consumer before the blocking call. I use Task.Run()here to start up a new thread. We will inside that thread Call the .GetConsumingEnumerable() to set up our consumer. Note that I also use Thread.Sleep to make the consumer wait a bit, this is only for testing and normally you would of course not wait here. Note also that you could spawn multiple consumers with Task.Run or similar to support 1 producer, multiple consumers scenarios. For multiple producers you could use a single shared blocking collection or possibly multiple blocking collections. BlockingCollection is very flexible. It is important that you call .CompleteAdding() on your producer side to allow the consumers to finish also their execution.
What if you want to not have a FIFO ordering in your Producer-Consumer scenario, but say a LIFO scenario? As you probably can see, if you instantiate the BlockingCollection, the constructor takes overloads to somethhing called an IProducerConsumerCollection Collection. So we just adjust our instantiation like the following (note that I had to remove the boundedCapacity here set to 1 to make it work!):

   private static readonly BlockingCollection<int> _blockingQueue = new BlockingCollection<int>(
            new ConcurrentStack<int>()); 

So now we get our new output:

Hit the any key!
Hey I got this item: 28!
Hey I got this item: 94!
Hey I got this item: 11!
Hey I got this item: 8!
Hey I got this item: 3!
Hey I got this item: 1!

The BCL contains several interesting classes in the System.Collections.Concurrent namespace. Happy .NET coding!

Tuesday, 21 June 2016

How to open up a SQL connection with SQL Management studio 2012 through Powershell

Sometimes it is nice to just log into a SQL server database with just running a command. To maintain security, we let our user input the password manually and use Powershell to start up SQL Server Management Studio 2012. If you got another version of SQL Server Management Server (SSMS), just adjust the path to ManagementStudio.

$domain="SOMEDOMAIN"
$user="SomeUser"
$workingdir="c:\Program Files (x86)\Microsoft SQL Server\110\Tools\Binn\ManagementStudio"
$databaseserver="somedbserver.somedomain.net"
$cmd="ssms.exe"
$arguments=" -S $databaseserver" 

$domainuser = $domain + "\" + $user

$response = Read-host "Enter password" -AsSecureString 
#$secpasswd = [Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($response))

$credential = New-Object System.Management.Automation.PSCredential ($domainuser, $response)

Start-Process -WorkingDirectory $workingdir -FilePath $cmd -Argument $arguments -Credential $credential 

You can add an icon on your desktop and assign it a shortcut to powershell and then paste the script above to a .ps1 file and then as the argument of the shortcut point to a .ps1 with the script above.

Wednesday, 1 June 2016

How to display the SQL involved for in Entity Framework programatically with DbContext and ObjectContext

As a developer, we often use an Object-Relational Mapper (ORM) to abstract from the way we work with a database. Gone are the days of building a SqLCommand object, setting the CommandText and executing row by row the result set, as used with ADO.NET. Today, most .NET developers use EntityFramework to work with the database. But this abstraction is all and well, and makes us work more efficient. Sadly, many developers are agnostic to the fact that despite they get the results out from the database, they do often do so in a slow manner. The reason of this is often not that there is a lot of data in the database, but we use Entity Framework queries that generate the wrong sql, i.e we get the results, but the SQL involved got a poor performance. You can use LinqPad for example to display the SQL involved in Entity Framework queries. But we can also achieve this using programatically methods, with C#. Here is an extension class I wrote to achieve this. The extension methods works with both data contexts that inherit from DbContext and data contexts that inherit from ObjectContext.

using System;
using System.Data.Entity.Core.Objects;
using System.Linq;
using System.Reflection;

namespace Hemit.OpPlan.Data.EntityFramework
{
    
    public static class IQueryableExtensions
    {

        /// <summary>
        /// Shows the sql the IQueryable query will be generated into and executed on the database
        /// </summary>
        /// <param name="query">The IQueryable to analyze</param>
        /// <param name="decodeParameters">Set to true if this method should try decoding the parameters</param>
        /// <remarks>This is the generated SQL query in use for Entity Framework. This works using ObjectContext</remarks>
        public static string ShowSqlUsingObjectContext(this IQueryable query, bool decodeParameters = false)
        {
            var objectQuery = (ObjectQuery)query; 
           
            string result = ((ObjectQuery)query).ToTraceString();

            if (!decodeParameters)
                return result; 

            foreach (var p in objectQuery.Parameters)
            {
                string valueString = p.Value != null ? p.Value.ToString() : string.Empty;
                if (p.ParameterType == typeof(string) || p.ParameterType == typeof(DateTime))
                    valueString = "'" + valueString + "'";
                result = result.Replace("@" +p.Name, p.Value != null ? valueString : string.Empty); 
            }
            return result; 
        }

        public static string ShowSqlUsingDbContext(this IQueryable query, bool decodeParameters = false)
        {
            var memberInfo = query.GetType().BaseType;
            if (memberInfo != null)
            {
                var internalQueryField = 
                memberInfo.GetFields(BindingFlags.NonPublic
              | BindingFlags.Instance).FirstOrDefault(f => f.Name.Equals("_internalQuery"));
                if (internalQueryField != null)
                {
                    var internalQuery = internalQueryField.GetValue(query);
                    var objectQueryField =
                        internalQuery.GetType().GetProperty("ObjectQuery"); 

                    // Here's your ObjectQuery!
                    if (objectQueryField != null)
                    {
                        var objectQuery = objectQueryField.GetValue(internalQuery) as ObjectQuery;
                        string sql = ShowSqlUsingObjectContext(objectQuery, decodeParameters);
                        return sql;
                    }
                }
            }

            return null;
        }

    }
}

Note that when we use an IQueryable inside a DbContext, the BaseType is actually a DbQuery, and this wraps the ObjectQuery inside a field called "_internalQuery". In addition, we get a property inside this field that is called "ObjectQuery". So we can get hold of the ObjectQuery inside a DbQuery. When we got hold of the ObjectQuery, it is easy to decode the contents using ToTraceString() method and if we want to further decode the parameters EntityFramework generates, we can do so using the Parameters property of ObjectQuery. We can then interpolate the SQL parametrization and get the SQL string most readable, if we like this form. Of course, some developers rather like the parametrized version. There may be some queries that don't have any parameters at all, but this is fine. Of course, all this parametrization business is to hinder SQL injection. Please do not resort to creating methods that accepts such "clean sql", you may easily generate an attack vector into your system if you try do adjusts queries so and not being careful. With this extension method we can easily test it out:

using SomeAcme.EntityFramework;
using Nunit.Framework; 

[Test]
public void TestGettingSomeSql(){

 using (var context = new SomeAcmeContext()){
  IQueryable query = context.SomeDataEntity.Where(x => x.SomeProperty == 123).AsQueryable();
  
  string sql = string.Empty; 

  //If this is a dbContext: 

   sql = query.ShowSqlUsingDbContext(decodeParameters: true);

   //Or if this is an ObjectContext: 

   sql = query.ShowSqlUsingObjectContext(decodeParameters: true); 

   Console.WriteLine(sql);

 }
}

We create a query, using the .AsQueryable() extension method in Linq and we then pass the IQueryable object to the extension methods of the class shown earlier. Of course, the extension method to use depends on the type of data context you work with. Either a DbContext or an ObjectContext.

Monday, 16 May 2016

How to ensure the integrity of information in .NET using Digital Signature Algorithm DSA

This article will concern the topic of digital signature. There are several ways to ensure the integrity of the information or data that is sent. This concerns the concept of non-repudiation, the case that the sender cannot deny that he or she is the true sender of that data. We also can check that the data is correct, so digital signature can act as some sort of checksum - but for the entire message. We are also concerned that the information is authentic and original and not tampered with by an attacker. Digital Signature Algorithm in .NET or DSA uses the SHA-1 Secure Hash Algorithm. There are today more powerful methods to sign data, such as the RSACryptoServiceProvider. But we will in this article use DSACryptoServiceProvider. DSA is today not considered failsafe. There are security vulnerabilities. However, for ordinary use - it is not that easy to break. Just like in RSA, there is a public and private key. The API is very similar to RSACryptoServiceProvider. The following console application shows some central API calls on a DSACryptoServiceProvider.

using System;
using System.Security.Cryptography;
using System.Text;

namespace DSASignDemo
{
    class Program
    {

        // ReSharper disable once UnusedParameter.Local
        static void Main(string[] args)
        {
            var dsa = new DSACryptoServiceProvider(1024);
            var publicDsaParameters = dsa.ExportParameters(false);
            var privateDsaParameters = dsa.ExportParameters(true);
            string inputText = "Burgers and coca cola";
            byte[] inputData = Encoding.Unicode.GetBytes(inputText);
            byte[] signedBytes = SignData(inputData, privateDsaParameters);
            bool isVerified = VerifyData(inputData, signedBytes, publicDsaParameters);

            Console.WriteLine("Input text: " + inputText);
            Console.WriteLine("Signed text: " + Convert.ToBase64String(signedBytes));

            if (isVerified)
                Console.WriteLine("The message was verified");
            else
                Console.WriteLine("The message was not verified");

            byte[] hashData = ComputeHash(inputData);
            Console.WriteLine("SHA-1 computed hash: " + Convert.ToBase64String(hashData));

            bool isHashSame = CompareHash(inputText, Convert.ToBase64String(hashData)); 
            if (isHashSame)
                Console.WriteLine("Hash is the same");
            else 
                Console.WriteLine("Hash is not same");

            byte[] signedHashData = dsa.SignHash(hashData, "SHA1");

            Console.WriteLine("Signed hash: ");
            Console.WriteLine(Convert.ToBase64String(signedHashData));

            bool isVerifiedHash = dsa.VerifyHash(hashData, "SHA1", signedHashData);
            if (isVerifiedHash)
                Console.WriteLine("Hash is verified");
            else
                Console.WriteLine("Hash is not verified");



            Console.WriteLine("Press the any key to continue ..");
            Console.ReadKey();  
        }

        static bool CompareHash(string inputText, string hashText)
        {
            string computedHash = Convert.ToBase64String(ComputeHash(Encoding.Unicode.GetBytes(inputText)));
            StringComparer comparer = StringComparer.OrdinalIgnoreCase;
            return comparer.Compare(computedHash, hashText) == 0; 
        }

        static byte[] ComputeHash(byte[] inputData)
        {
            var shaManaged = new SHA1Managed();
            byte[] hashBytes = shaManaged.ComputeHash(inputData);
            return hashBytes;
        }

        static byte[] SignData(byte[] inputData, DSAParameters dsaParameters)
        {
            try
            {
                var dsa = new DSACryptoServiceProvider();
                dsa.ImportParameters(dsaParameters);
                return dsa.SignData(inputData);
            }
            catch (CryptographicException cge)
            {
                Console.WriteLine(cge.Message);
                return null;
            }
        }

        static bool VerifyData(byte[] inputData, byte[] signedData, DSAParameters dsaParmeters)
        {
            try
            {
                var dsa = new DSACryptoServiceProvider();
                dsa.ImportParameters(dsaParmeters);
                return dsa.VerifyData(inputData, signedData); 
            }
            catch (Exception err)
            {
                Console.WriteLine(err.Message);
                return false;
            }
        }




    }
}


A sample output of running this console application is:
 


Input text: Burgers and coca cola
Signed text: su7Qv+O58MyOzFjWXXx6bq9xAz9GtJ30+N8pmEYA4qFwmCdU04+qWg==
The message was verified
SHA-1 computed hash: b4o//84sCZ5cUY6cfewNia9yHYI=
Hash is the same
Signed hash:
xWExD3udQWayE2nfVDY+w8o/VuuBlKRng5Oe5XZ1zBAJO90BG+dbcA==
Hash is verified
Press the any key to continue ..


Note that the output will differ per run, where it says signed text and SHA-1 computed hash and signed hash. The reason is that the DSA algorithm will choose a random number in part of its steps and the resulting output will give different results. The boolean values here should of course be consistent, i.e. give consistens checks. Some things to note here: - Using the constructor we ask for at least a 1024 bit sized BigNum in the constructor of the DSACryptoServiceProvider to be used to generate the large primes that is involved in the DSA algorithm. - We actually use the private key of DSA to sign data and the public key to verify the data. DSA is an assymetric cryptographic algoritm and the order in which the keys are used is kind of reversed to RSA. It is the sender that sign the data and the receiver that verifies the data with a public key. - For speed, we can sometimes choose to just compute a hash like SHA-1 and then sign this hash. We can then verify hash. This is much quicker than signing large data. So first off, we can compute a SHA-1 hash, then sign the hash and then include this signed hash appended to the message, then let the receiver just verify the signed hash. The receiver will then use the hash and the signed hash and verify the hash and the fact that the message integrity is kept. We must tell the method VerifyHash which algorithm that is used. An overview of the hash algorithm names you can use in one of the arguments of SignHash and VerifyHash methods is available here: https://msdn.microsoft.com/en-us/library/system.security.cryptography.hashalgorithmname(v=vs.110)

How to do async calls without locking the UI thread in WPF

WPF developers that have worked with async await have most likely run into problems with avoiding race conditions with the UI thread, either making the entire UI lock up or burden the UI thread and cause clients not responding. This article will show you how to avoid this. The way to do this is to await using another thread and afterwards use that result back again on the WPF thread to do the updates in the UI. We use ThreadPool.QueueUserWorkItem for this. As you can see, we user an inner method marked with async keyword to await the results. This is done to avoid the classic "async proliferation" seen in async code, where the async keyword spreads upwards to all methods. We instead use an outher method and call the async method and use the Result of the Task returned. We could do some quality checking here, to check if the Task succeeded of course. The Task object contains status information about if the results are really available or not and Result will throw an exception if there was an error in the retrieval of the async results from the lower layers of the software app layers. Example code:

DispatcherUtil.AsyncWorkAndUiThreadUpdate(Dispatcher.CurrentDispatcher, () => GetSomeItems(someId),
 x => GetSomeItemsUpdateUIAfterwards(x), displayWaitCursor:true);
//Note here that we retrieve the data not on the UI thread, but on a dedicated thread and after retrieved the
//result, we do an update in the GUI. 
private List<SomeItemDataContract> GetSomeItems(int someId)
        {
         var retrieveTask = GomeSomeItemsInnerAsync(someId);
         return retrieveTask.Result;
        }
 
private async Task<List<SomeItemDataContract>> GetSomeItemsInnerAsync(int someId)
        {
         List<SomeItemDataContract> retrieveTask = await SomeServiceAgent.GetSomeItems(someId);
         return retrieveTask;
        }

private void GetSomeItemsUpdateUIAfterwards(SomeItemDataContract x){
 if (x != null){
  //Do some UI stuff - remember RaisePropertyChanged
 }
}


Utility method:

public static void AsyncWorkAndUiThreadUpdate<T>(Dispatcher currentDispatcher, Func<T> threadWork, Action<T> guiUpdate, 
bool displayWaitCursor = false)
        {
         if (displayWaitCursor)
          PublishMouseCursorEvent<T>(Cursors.Wait);

         // ReSharper disable once UnusedAnonymousMethodSignature 
         ThreadPool.QueueUserWorkItem(delegate(object state)
            {
              T resultAfterThreadWork = threadWork();
              // ReSharper disable once UnusedAnonymousMethodSignature
              currentDispatcher.BeginInvoke(DispatcherPriority.Normal, new Action<T>(delegate {
       
              if (displayWaitCursor)
               PublishMouseCursorEvent<T>(Cursors.Arrow);
 
              guiUpdate(resultAfterThreadWork);
           }), resultAfterThreadWork);

            });
 
        }

The PublishMouseCursorEvent publishes a prism event that is captured by a Bootstrapper class, but what you choose to do here is of course up to you. One way is to subscribe such an event (either a CompositePresentationEvent as in Prism or an ordinary CLR event for example):
private void OnCursorEvent(CursorEventArg eventArg)
{
 if (eventArg != null)
 {
 Mouse.OverrideCursor = eventArg.Cursor;
 }
}