Showing posts with label .net. Show all posts
Showing posts with label .net. Show all posts

Sunday 31 December 2023

AES Encryption with Galois Counter Mode (GCM) in C#

This article presents some helper methods for performing AES Encryption using Galois Counter Mode (GCM). AES or Advanced Encryption Standard is the most used encryption algorithm used today, having overtaken DES and Triple DES since 2001. We will look into the GCM mode of AES in this article. AES-GCM class AesGcm is supported in .NET Core 3.0 and newer .NET versions, plus in .NET Standard 2.1. AES-GCM is authenticated encryption, compared to default AES-CBC (Cipher Block Chaining). Benefits of using GCM mode of AES is the following:
  • Data authenticity / integrity. This is provided via a tag that is outputted by the encryption and used while decrypting
  • Provides support for sending additional data, used for example in newer TLS implementations to provide both encryption and a non-encrypted payload. This is called additional metadata
Here is a helper class to perform encryption and decryption using AES-GCM.
 
 public static class AesGcmEncryption {


	public static (byte[], byte[]) Encrypt(byte[] dataToEncrypt, byte[] key, byte[] nonce, byte[] associatedData = null)
	{
		using var aesGcm = new AesGcm(key);
		//tag and ciphertext will be filled during encryption
		var tag = new byte[16]; //tag is a hmac (hash-based message authentication code) to check that information has not been tampered with
	    var cipherText = new byte[dataToEncrypt.Length];
		aesGcm.Encrypt(nonce, dataToEncrypt, cipherText, tag, associatedData);
		return (cipherText, tag);
	}

	public static byte[] Decrypt(byte[] cipherText, byte[] key, byte[] nonce, byte[] tag, byte[] associatedData = null)
	{
		using var aesGcm = new AesGcm(key);
		//tag and ciphertext will be filled during encryption
		var decryptedData = new byte[cipherText.Length];
		aesGcm.Decrypt(nonce, cipherText, tag, decryptedData, associatedData);
		return decryptedData;
	}
	
}
 
 
In the code above, the encrypt method returns a tuple with the ciperText and the tag. These are the encrypted data and the tag, both must be used while decrypting and the tag provides as mentioned a means of checking the integrity of data, i.e. that data has not been tampered with. Note that the 16-byte tag and the ciphertext is filled after running the Encrypt method of the AesGcm class. The cipherText array must be the same length as the dataToEncrypt array inputted. Here is sample code to use AES-GCM. Note that the metadata used here, while optional, must match in case it is set in the encryption and decryption. The nonce must be 12 bytes - 96 bits in length.The nonce is similar to a initialization vector, although it is used once for the particular encryption and decryption, it is used to protect against replay attacks.
 
 
 void TestAesGCM()
{
	const string original = "Text to encrypt";
	var key = RandomNumberGenerator.GetBytes(32); //256 bits key
	var nonce = RandomNumberGenerator.GetBytes(12); //96 bits nonce
	
	(byte[] cipherText, byte[] tag) result = AesGcmEncryption.Encrypt(Encoding.UTF8.GetBytes(original),
	 key, nonce, Encoding.UTF8.GetBytes("some metadata 123"));
	 byte[] decryptedText = AesGcmEncryption.Decrypt(result.cipherText, key, nonce, result.tag, Encoding.UTF8.GetBytes("some metadata 123")); 
		
	Console.WriteLine("AES Encryption demo GCM - Galois Counter Mode:");
	Console.WriteLine("--------------");
	Console.WriteLine("Original Text = " + original);
	Console.WriteLine("Encrypted Text = " + Convert.ToBase64String(result.cipherText));
	Console.WriteLine("Tag = " + Convert.ToBase64String(result.tag));
	Console.WriteLine("Decrypted Text = " + Encoding.UTF8.GetString(decryptedText));
}
 
 
AES Encryption demo GCM - Galois Counter Mode: -------------- Original Text = Text to encrypt Encrypted Text = 9+2x0kctnRwiDDHBm0/H Tag = sSDxsg17HFdjE4cuqRlroQ== Decrypted Text = Text to encrypt Use AES-GCM to provide integrity checking and allowing to send in metadata if desired to encrypt and decrypting with the AES algorithm. We can protect the AES key using different methods, for example using the Data Protection API, this is only supported in Windows. Let's look at a helper class for using Data Protection API.
 
 
 public static class DataProtectionUtil {

	public static byte[] Protect(byte[] dataToEncrypt, byte[] optionalEntropy, DataProtectionScope scope)
	{
		var encryptedData = ProtectedData.Protect(dataToEncrypt, optionalEntropy, scope);
		return encryptedData;
	}
	
	public static byte[] Unprotect(byte[] encryptedData, byte[] optionalEntropy, DataProtectionScope scope){
		var decryptedData = ProtectedData.Unprotect(encryptedData, optionalEntropy, scope);
		return decryptedData;
	}

	public static string Protect(string dataToEncrypt, string optionalEntropy, DataProtectionScope scope)
	{
		var encryptedData = ProtectedData.Protect(Encoding.UTF8.GetBytes(dataToEncrypt), optionalEntropy != null ? Encoding.UTF8.GetBytes(optionalEntropy) : null, scope);
		return Convert.ToBase64String(encryptedData);
	}

	public static string Unprotect(string encryptedData, string optionalEntropy, DataProtectionScope scope)
	{
		var decryptedData = ProtectedData.Unprotect(Convert.FromBase64String(encryptedData), optionalEntropy != null ? Encoding.UTF8.GetBytes(optionalEntropy) : null, scope);
		return Encoding.UTF8.GetString(decryptedData);
	}

}
 
 

An example how to protect your AES key:

 
 
void EncryptAndDecryptWithProtectedKey(){
	var original = "Text to encrypt";
	Console.WriteLine($"Original Text = {original}");
	
	//Create key and nnoce . Encrypt our text with AES 
	var gcmKey = RandomNumberGenerator.GetBytes(32);
	var nonce = RandomNumberGenerator.GetBytes(12); 
	
	var result = EncryptText(original, gcmKey, nonce); 
	
	//Create some entropy and protect AES key
	var entropy = RandomNumberGenerator.GetBytes(16); 
	var protectedKey = ProtectedData.Protect(gcmKey, entropy, DataProtectionScope.CurrentUser);

	Console.WriteLine($"gcmKey = {Convert.ToBase64String(gcmKey)}, protectedKey = {Convert.ToBase64String(protectedKey)}");
	
	// Decrypt the text with AES. the AES key has to be retrieved with DPAPI.
	var decryptedText = DecryptText(result.encrypted, nonce, result.tag, protectedKey, entropy);

	Console.WriteLine($"Decrypted Text using AES GCM with key retrieved via Data Protection API = {decryptedText}");

}

private static (byte[] encrypted, byte[] tag) EncryptText(string original, byte[] gcmKey, byte[] nonce){
	return AesGcmEncryption.Encrypt(Encoding.UTF8.GetBytes(original), gcmKey, nonce, Encoding.UTF8.GetBytes("some meta"));
}

private static string DecryptText(byte[] encrypted, byte[] nonce, byte[] tag, byte[] protectedKey, byte[] entropy){
	
	var key = DataProtectionUtil.Unprotect(protectedKey, entropy, DataProtectionScope.CurrentUser);

	Console.WriteLine($"Inside DecryptText: gcmKey = {Convert.ToBase64String(key)}, protectedKey = {Convert.ToBase64String(protectedKey)}");

	var decryptedText = AesGcmEncryption.Decrypt(encrypted, key, nonce, tag, Encoding.UTF8.GetBytes("some meta"));
	return Encoding.UTF8.GetString(decryptedText);
}
 
Data Protection API is only supported on Windows platform, there are more possibilities to protect AES key but protecting your key is always a challenge when dealing with symmetric encryption algorithms such as AES. Some more links:

Friday 22 September 2023

Using Azure Computer Vision to perform Optical Character Recognition (OCR)

This article shows how you can use Azure Computer vision in Azure Cognitive Services to perform Optical Character Recognition (OCR). The Computer vision feature is available by adding a Computer Vision resource in Azure Portal. I have made a .NET MAUI Blazor app and the Github Repo for it is available here : https://github.com/toreaurstadboss/Ocr.Handwriting.Azure.AI.Models
Let us first look at the .csproj of the Lib project in this repo.


<Project Sdk="Microsoft.NET.Sdk.Razor">

  <PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>
  <ItemGroup>
    <SupportedPlatform Include="browser" />
  </ItemGroup>

	<ItemGroup>
		<PackageReference Include="Microsoft.Azure.CognitiveServices.Vision.ComputerVision" Version="7.0.1" />
		<PackageReference Include="Microsoft.AspNetCore.Components.Web" Version="6.0.19" />
	</ItemGroup>

</Project>


The following class generates ComputerVision clients that can be used to extract different information from streams and files containing video and images. We are going to focus on images and extracting text via OCR. Azure Computer Vision can also extract handwritten text in addition to regular text written by typewriters or text inside images and similar. Azure Computer Vision also can detect shapes in images and classify objects. This demo only focuses on text extraction form images. ComputerVisionClientFactory


using Microsoft.Azure.CognitiveServices.Vision.ComputerVision;

namespace Ocr.Handwriting.Azure.AI.Lib
{

    public interface IComputerVisionClientFactory
    {
        ComputerVisionClient CreateClient();
    }

    /// <summary>
    /// Client factory for Azure Cognitive Services - Computer vision.
    /// </summary>
    public class ComputerVisionClientFactory : IComputerVisionClientFactory
    {
        // Add your Computer Vision key and endpoint
        static string? _key = Environment.GetEnvironmentVariable("AZURE_COGNITIVE_SERVICES_VISION_KEY");
        static string? _endpoint = Environment.GetEnvironmentVariable("AZURE_COGNITIVE_SERVICES_VISION_ENDPOINT");

        public ComputerVisionClientFactory() : this(_key, _endpoint)
        {
        }

        public ComputerVisionClientFactory(string? key, string? endpoint)
        {
            _key = key;
            _endpoint = endpoint;
        }

        public ComputerVisionClient CreateClient()
        {
            if (_key == null)
            {
                throw new ArgumentNullException(_key, "The AZURE_COGNITIVE_SERVICES_VISION_KEY is not set. Set a system-level environment variable or provide this value by calling the overloaded constructor of this class.");
            }
            if (_endpoint == null)
            {
                throw new ArgumentNullException(_key, "The AZURE_COGNITIVE_SERVICES_VISION_ENDPOINT is not set. Set a system-level environment variable or provide this value by calling the overloaded constructor of this class.");
            }

            var client = Authenticate(_key!, _endpoint!);
            return client;
        }

        public static ComputerVisionClient Authenticate(string key, string endpoint) =>
            new ComputerVisionClient(new ApiKeyServiceClientCredentials(key))
            {
                Endpoint = endpoint
            };

    }
}



The setup of the endpoint and key of the Computer Vision resource is done via system-level envrionment variables. Next up, let's look at retrieving OCR text from images. Here we use ComputerVisionClient. We open up a stream of a file, an image, using File.OpenReadAsync and then the method ReadInStreamAsync of Computer vision client. The image we will load up in the app is selected by the user and the image is previewed and saved using MAUI Storage lib (inside the Appdata folder). OcrImageService.cs


using Microsoft.Azure.CognitiveServices.Vision.ComputerVision;
using Microsoft.Azure.CognitiveServices.Vision.ComputerVision.Models;
using Microsoft.Extensions.Logging;
using System.Diagnostics;
using ReadResult = Microsoft.Azure.CognitiveServices.Vision.ComputerVision.Models.ReadResult;

namespace Ocr.Handwriting.Azure.AI.Lib
{

    public interface IOcrImageService
    {
        Task<IList<ReadResult?>?> GetReadResults(string imageFilePath);
        Task<string> GetReadResultsText(string imageFilePath);
    }

    public class OcrImageService : IOcrImageService
    {
        private readonly IComputerVisionClientFactory _computerVisionClientFactory;
        private readonly ILogger<OcrImageService> _logger;

        public OcrImageService(IComputerVisionClientFactory computerVisionClientFactory, ILogger<OcrImageService> logger)
        {
            _computerVisionClientFactory = computerVisionClientFactory;
            _logger = logger;
        }

        private ComputerVisionClient CreateClient() => _computerVisionClientFactory.CreateClient();

        public async Task<string> GetReadResultsText(string imageFilePath)
        {
            var readResults = await GetReadResults(imageFilePath);
            var ocrText = ExtractText(readResults?.FirstOrDefault());
            return ocrText;
        }

        public async Task<IList<ReadResult?>?> GetReadResults(string imageFilePath)
        {
            if (string.IsNullOrWhiteSpace(imageFilePath))
            {
                return null;
            }

            try
            {
                var client = CreateClient();

                //Retrieve OCR results 

                using (FileStream stream = File.OpenRead(imageFilePath))
                {
                    var textHeaders = await client.ReadInStreamAsync(stream);
                    string operationLocation = textHeaders.OperationLocation;
                    string operationId = operationLocation[^36..]; //hat operator of C# 8.0 : this slices out the last 36 chars, which contains the guid chars which are 32 hexadecimals chars + four hyphens

                    ReadOperationResult results;

                    do
                    {
                        results = await client.GetReadResultAsync(Guid.Parse(operationId));
                        _logger.LogInformation($"Retrieving OCR results for operationId {operationId} for image {imageFilePath}");
                    }
                    while (results.Status == OperationStatusCodes.Running || results.Status == OperationStatusCodes.NotStarted);

                    IList<ReadResult?> result = results.AnalyzeResult.ReadResults;
                    return result;

                }
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.Message);
                return null;
            }
        }

        private static string ExtractText(ReadResult? readResult) => string.Join(Environment.NewLine, readResult?.Lines?.Select(l => l.Text) ?? new List<string>());

    }

}
                                           

Let's look at the MAUI Blazor project in the app. The MauiProgram.cs looks like this. MauiProgram.cs


using Ocr.Handwriting.Azure.AI.Data;
using Ocr.Handwriting.Azure.AI.Lib;
using Ocr.Handwriting.Azure.AI.Services;
using TextCopy;

namespace Ocr.Handwriting.Azure.AI;

public static class MauiProgram
{
    public static MauiApp CreateMauiApp()
    {
        var builder = MauiApp.CreateBuilder();
        builder
            .UseMauiApp<App>()
            .ConfigureFonts(fonts =>
            {
                fonts.AddFont("OpenSans-Regular.ttf", "OpenSansRegular");
            });

        builder.Services.AddMauiBlazorWebView();
#if DEBUG
        builder.Services.AddBlazorWebViewDeveloperTools();
        builder.Services.AddLogging();
#endif

        builder.Services.AddSingleton<WeatherForecastService>();
        builder.Services.AddScoped<IComputerVisionClientFactory, ComputerVisionClientFactory>();
        builder.Services.AddScoped<IOcrImageService, OcrImageService>();
        builder.Services.AddScoped<IImageSaveService, ImageSaveService>();

        builder.Services.InjectClipboard();

        return builder.Build();
    }
}



We also need some code to preview and save the image an end user chooses. The IImageService looks like this. ImageSaveService


using Microsoft.AspNetCore.Components.Forms;
using Ocr.Handwriting.Azure.AI.Models;

namespace Ocr.Handwriting.Azure.AI.Services
{

    public class ImageSaveService : IImageSaveService
    {

        public async Task<ImageSaveModel> SaveImage(IBrowserFile browserFile)
        {
            var buffers = new byte[browserFile.Size];
            var bytes = await browserFile.OpenReadStream(maxAllowedSize: 30 * 1024 * 1024).ReadAsync(buffers);
            string imageType = browserFile.ContentType;

            var basePath = FileSystem.Current.AppDataDirectory;
            var imageSaveModel = new ImageSaveModel
            {
                SavedFilePath = Path.Combine(basePath, $"{Guid.NewGuid().ToString("N")}-{browserFile.Name}"),
                PreviewImageUrl = $"data:{imageType};base64,{Convert.ToBase64String(buffers)}",
                FilePath = browserFile.Name,
                FileSize = bytes / 1024,
            };

            await File.WriteAllBytesAsync(imageSaveModel.SavedFilePath, buffers);

            return imageSaveModel;
        }

    }
}


Note the use of maxAllowedSize of IBrowserfile.OpenReadStream method, this is a good practice since IBrowserFile only supports 512 kB per default. I set it in the app to 30 MB to support some high res images too. We preview the image as base-64 here and we also save the image also. Note the use of FileSystem.Current.AppDataDirectory as base path here. Here we use nuget package Microsoft.Maui.Storage. These are the packages that is used for the MAUI Blazor project of the app. Ocr.Handwriting.Azure.AI.csproj



    <ItemGroup>
      <PackageReference Include="Microsoft.Azure.CognitiveServices.Vision.ComputerVision" Version="7.0.1" />
      <PackageReference Include="TextCopy" Version="6.2.1" />
    </ItemGroup>


The GUI looks like this : Index.razor


@page "/"
@using Ocr.Handwriting.Azure.AI.Models;
@using Microsoft.Azure.CognitiveServices.Vision.ComputerVision;
@using Microsoft.Azure.CognitiveServices.Vision.ComputerVision.Models;
@using Ocr.Handwriting.Azure.AI.Lib;
@using Ocr.Handwriting.Azure.AI.Services;
@using TextCopy;

@inject IImageSaveService ImageSaveService
@inject IOcrImageService OcrImageService 
@inject IClipboard Clipboard

<h1>Azure AI OCR Text recognition</h1>


<EditForm Model="Model" OnValidSubmit="@Submit" style="background-color:aliceblue">
    <DataAnnotationsValidator />
    <label><b>Select a picture to run OCR</b></label><br />
    <InputFile OnChange="@OnInputFile" accept=".jpeg,.jpg,.png" />
    <br />
    <code class="alert-secondary">Supported file formats: .jpeg, .jpg and .png</code>
    <br />
    @if (Model.PreviewImageUrl != null) { 
        <label class="alert-info">Preview of the selected image</label>
        <div style="overflow:auto;max-height:300px;max-width:500px">
            <img class="flagIcon" src="@Model.PreviewImageUrl" /><br />
        </div>

        <code class="alert-light">File Size (kB): @Model.FileSize</code>
        <br />
        <code class="alert-light">File saved location: @Model.SavedFilePath</code>
        <br />

        <label class="alert-info">Click the button below to start running OCR using Azure AI</label><br />
        <br />
        <button type="submit">Submit</button> <button style="margin-left:200px" type="button" class="btn-outline-info" @onclick="@CopyTextToClipboard">Copy to clipboard</button>
        <br />
        <br />
        <InputTextArea style="width:1000px;height:300px" readonly="readonly" placeholder="Detected text in the image uploaded" @bind-Value="Model!.OcrOutputText" rows="5"></InputTextArea>
    }
</EditForm>


@code {

    private IndexModel Model = new();

    private async Task OnInputFile(InputFileChangeEventArgs args)
    {       
        var imageSaveModel = await ImageSaveService.SaveImage(args.File);
        Model = new IndexModel(imageSaveModel);
        await Application.Current.MainPage.DisplayAlert($"MAUI Blazor OCR App", $"Wrote file to location : {Model.SavedFilePath} Size is: {Model.FileSize} kB", "Ok", "Cancel");
    }

    public async Task CopyTextToClipboard()
    {
        await Clipboard.SetTextAsync(Model.OcrOutputText);
        await Application.Current.MainPage.DisplayAlert($"MAUI Blazor OCR App", $"The copied text was put into the clipboard. Character length: {Model.OcrOutputText?.Length}", "Ok", "Cancel");

    }

    private async Task Submit()
    {
        if (Model.PreviewImageUrl == null || Model.SavedFilePath == null)
        {
            await Application.Current.MainPage.DisplayAlert($"MAUI Blazor OCR App", $"You must select an image first before running OCR. Supported formats are .jpeg, .jpg and .png", "Ok", "Cancel");
            return;
        }
        Model.OcrOutputText = await OcrImageService.GetReadResultsText(Model.SavedFilePath);
        StateHasChanged(); //visual refresh here
    }

}


The UI works like this. The user selects an image. As we can see by the 'accept' html attribute, the .jpeg, .jpg and .png extensions are allowed in the file input dialog. When the user selects an image, the image is saved and previewed in the UI. By hitting the Submit button, the OCR service in Azure is contacted and text is retrieved and displayed in the text area below, if any text is present in the image. A button allows copying the text into the clipboard. Here are some screenshots of the app.


Saturday 22 April 2023

Tag Helpers in Asp.net Core Mvc 7

This article will present a sample Tag Helper in .net. A Tag Helper is similar to Html Helpers in Asp.net Mvc in .NET Framework, but it is easier to use in HTML as it does not use the special "@-syntax". The Tag helper will render a list using the <ul> and <li> tags. In addition, Bootstrap 5 will be used. Start by creating a razor application with this command:
dotnet new razor -o TagHelpers Then move into the folder TagHelpers and type: code .

Inside Visual Studio Code, hit Ctrl+P and look up the file _ViewImports.cshtml and add the current assembly/solution using:

@addTagHelper *, TagHelpers

This tells that we want to add any TagHelper from the assembly called TagHelpers (the solution we are working with).

@using TagHelpers
@namespace TagHelpers.Pages
@addTagHelper *, Microsoft.AspNetCore.Mvc.TagHelpers
@addTagHelper *, TagHelpers

Consider the following HTML :

<list separator="|">option 1| option 2| option 3| option 4| option 5| option 6| option 7| option 8|this is fun<list>
We want to turn that HTML into the list shown in screen shot below :
That is - create a list using an <ul> tag followed by <li> tags inside. Since we need to access the inner content of the HTML here, we have to use ProcessAsync method of derived method from the TagHelper. We create a TagHelper by inheriting from this class and we also have to name the class suffixed by TagHelper by convention. The resulting Tag Helper then looks like this:


using System.Text;
using Microsoft.AspNetCore.Razor.TagHelpers;

namespace TagHelpers.TagHelpers;

public class ListTagHelper : TagHelper {

    public override async Task ProcessAsync(TagHelperContext context, TagHelperOutput output)
    {
        output.TagName = "ul";
        output.Attributes.Add("class", "list-group");
        output.Attributes.Add("style", "display:inline-block"); 
        var existingContent = await output.GetChildContentAsync(); 
        var allContent = existingContent.GetContent();
        var items = allContent.Trim().Split(new[] { Separator }, StringSplitOptions.None);
        var outputHtml = new StringBuilder();
        foreach (var item in items){
            outputHtml.Append($@"<li class=""list-group-item"">{item}</li>");
        }
        output.Content.SetHtmlContent(outputHtml.ToString());        
    }
    public string Separator { get; set; } = ",";
}


We default set the property Separator default to "," to separate items in our list. We could use another separator, such as "|" shown in the markup. If you omit the Separator, "," will be default used. Each public property becomes a recognized attribute in your TagHelper and can be used in the HTML. The TagName is the tag that will be used in the HTML. As we see, we also add 'class' and 'style' attributes here to show a list-group in HTML using Bootstrap 5 classes. We also split the items using the separator, make not that we use the GetChildContentAsync() method on the TagHelperOutput output object, followed by GetContent() method call. Also note that we have to use SetHtmlContent method in case we want to add explicit html content in the content of our 'a' tag here. It is suggested that you stick to string properties in Razor tag helpers instead of other data types.

Monday 15 July 2019

Using Ndepend to investigate method cycles

Ndepend is a very powerful Code analysis platform for DotNet and I have created a method cycle detection rule. The method cycle detection also supports property setters and getters. A cycle in a property or method will often cause problems, such as stack overflow exceptions due to too many recursive calls. Spotting such cycles can often be hard in real-world scenarios. A classic error is to actually do a cycle through a property getter, by just calling the getter once more or a property setter for that instance. The following C# attribute invokes a Rules extracted from Source code from Ndepend.

using System;
using NDepend.Attributes;

namespace Hemit.OpPlan.Client.Infrastructure.Aspects
{
    /// Ndepend attribute to enable cyclic loops in methods in the source code 
    [CodeRule(@"// <Name>Avoid methods of a type to be in cycles. Detect also recursive property setter calls</Name>
warnif count > 0

from t in Application.Types
                 .Where(t => t.ContainsMethodDependencyCycle != null && 
                             t.ContainsMethodDependencyCycle.Value)

// Optimization: restreint methods set
// A method involved in a cycle necessarily have a null Level.
let methodsSuspect = t.Methods.Where(m => m.Level == null)

// hashset is used to avoid iterating again on methods already caught in a cycle.
let hashset = new HashSet<IMethod>()


from suspect in methodsSuspect
   // By commenting this line, the query matches all methods involved in a cycle.
   where !hashset.Contains(suspect)

   // Define 2 code metrics
   // - Methods depth of is using indirectly the suspect method.
   // - Methods depth of is used by the suspect method indirectly.
   // Note: for direct usage the depth is equal to 1.
   let methodsUserDepth = methodsSuspect.DepthOfIsUsing(suspect)
   let methodsUsedDepth = methodsSuspect.DepthOfIsUsedBy(suspect)

   // Select methods that are both using and used by methodSuspect
   let usersAndUsed = from n in methodsSuspect where 
                         methodsUserDepth[n] > 0 && 
                         methodsUsedDepth[n] > 0 
                      select n

   where usersAndUsed.Count() > 0

   // Here we've found method(s) both using and used by the suspect method.
   // A cycle involving the suspect method is found!
   let cycle = System.Linq.Enumerable.Append(usersAndUsed,suspect)


   // Fill hashset with methods in the cycle.
   // .ToArray() is needed to force the iterating process.
   let unused1 = (from n in cycle let unused2 = hashset.Add(n) select n).ToArray()

let cycleContainsSetter = (from n1 in cycle where n1.IsPropertySetter select n1).Count()

  
select new { suspect, cycle, cycleContainsSetter }",
        Active = true,
        DisplayStatInReport = true,
        DisplayListInReport = true,
        DisplaySelectionViewInReport = true,
        IsCriticalRule = false)]
    public class MethodCycleDetectionAttribute : Attribute
    {
    }
}


Note that this Ndepend Code rule uses the CQLinq syntax placed into a C# attribute class that inherits from Attribute and itself is attributed with a Ndepend.Attributes.CodeRuleAttribute. I had to adjust the attribute a little bit using the ExtensionMethodsEnumerable fix for .Net 4.6.2 mentioned on Ndepend blog post here: https://blog.ndepend.com/problem-extension-methods/ The following screen shot shows Visual Studio 2019 with Ndepend version 2019.2.5 installed. It shows the method cycle detection code rule I created showing up as a code rule through source code. Selecting that code rule in the menu Ndepend => Rules Explorer that selects Queries and Rules Explorer allows to not only view the code rule but quickly see the code analysis results.
One method cycle I have is shown in the following image, the method ProcessPasRequest of my system. This case shows how advanced Ndepend really is in code analysis. It can detect transitive cycles with multiple function calls. The following two screen shots shows the cycle in question. In this case, it was not a bug, since the cycle will only happend a finite amount of times for a given set of patients, but it still can go into an infinite loop if the external system is not working correct. However in this case, if that external system is not working, the system I have been working with also faults as it relies on that external system. With Ndepend I could spot method cycles with transitive cycles with a breeze! I can also spot property getter and setter cycles, as property getters and setters in C# are methods anyways (compiler generated). I managed to get Ndepend back to the main menu in Visual Studio 2019 using this extension.
https://marketplace.visualstudio.com/items?itemName=Evgeny.RestoreExtensions

Thursday 2 May 2019

Regenerating precompiled views in Entity Framework

This article will present a solution to regenerate precompiled views in Entity Framework. Precompiled views can have a dramatic effect on the startup time of your DbContext / ObjectContext, especially the time to execute the first query against the database. First off, the following class can generate these precompiled views:

using System;
using System.Collections.Generic;
using System.Data.Entity.Core.Mapping;
using System.Data.Entity.Core.Metadata.Edm;
using System.Data.Entity.Infrastructure;

namespace PrecompiledViewGenerator
{
    /// <summary>
    /// Capable of generating pre compiled views that EF can use for quicker startup time when application domains starts.
    /// </summary>
    /// <typeparam name="TDbContext"></typeparam>
    /// <param name="dbContext"></param>
    /// <remarks>See https://github.com/ErikEJ/EntityFramework6PowerTools/tree/community/src/PowerTools Github page for the source code of EF Power tools.</remarks>
    /// <returns>A string containing the precompiled views that can be written to file or database for later use to gain optimized startup speeds of EF in application domains.</returns>
    public class EntityFrameworkPrecompiledViewGenerator
    {
        /// <summary>
        /// Generates pre compiled views from a db context. Uses EF Powertools runtime T4 template (.tt file) for precompiled view generation of views for EF 6 ObjectContext for C#.
        /// </summary>
        /// <typeparam name="TDbContext"></typeparam>
        /// <param name="dbContext"></param>
        /// <param name="viewContainerSuffix">The suffix to apply in the generated file containing the precompiled views</param>
        /// <remarks>See https://github.com/ErikEJ/EntityFramework6PowerTools/tree/community/src/PowerTools Github page for the source code of EF Power tools.</remarks>
        /// <returns>A string containing the precompiled views that can be written to file or database for later use to gain optimized startup speeds of EF in application domains.</returns>
        public string GeneratePrecompiledViews<TDbContext>(TDbContext dbContext, string viewContainerSuffix) where TDbContext : IObjectContextAdapter
        {
            if (string.IsNullOrEmpty(viewContainerSuffix))
                throw new ArgumentNullException(nameof(viewContainerSuffix));
            var objectContext = (dbContext as IObjectContextAdapter)?.ObjectContext; 
            if (objectContext == null)
                throw new ArgumentNullException(nameof(dbContext));
            var viewGenerator = new CSharpViewGenerator();
            var mappingCollection = (StorageMappingItemCollection) objectContext.MetadataWorkspace.GetItemCollection(DataSpace.CSSpace);
            if (mappingCollection == null)
                throw new ArgumentNullException(nameof(dbContext));
            var listOfEdmSchemaErrors = new List<EdmSchemaError>();
            var views = mappingCollection.GenerateViews(listOfEdmSchemaErrors);
            listOfEdmSchemaErrors.ForEach(error =>
            {
                if (error.Severity == EdmSchemaErrorSeverity.Error)
                    throw new ArgumentOutOfRangeException($"An error occurred while trying to generate views for {dbContext.GetType().Name}. The error was: {error.ToString()}");
            });
            viewGenerator.ContextTypeName = dbContext.GetType().FullName;
            viewGenerator.MappingHashValue = mappingCollection.ComputeMappingHashValue();
            viewGenerator.ViewContainerSuffix = viewContainerSuffix;
            viewGenerator.Views = views;
            string precompiledViews = viewGenerator.TransformText();
            return precompiledViews;
        }
    }
}



The following runtime text template (.T4) is used to generate the precompiled views:

//------------------------------------------------------------------------------
// <auto-generated>
//     This code was generated by a tool.
//
//     Changes to this file may cause incorrect behavior and will be lost if
//     the code is regenerated.
// </auto-generated>
//------------------------------------------------------------------------------

using System.Data.Entity.Infrastructure.MappingViews;

[assembly: DbMappingViewCacheTypeAttribute(
    typeof(<#= ContextTypeName #>),
    typeof(Edm_EntityMappingGeneratedViews.ViewsForBaseEntitySets<#= ContextTypeName #>))]

namespace Edm_EntityMappingGeneratedViews
{
    using System;
    using System.CodeDom.Compiler;
    using System.Data.Entity.Core.Metadata.Edm;

    /// <summary>
    /// Implements a mapping view cache.
    /// </summary>
    [GeneratedCode("Entity Framework 6 Power Tools", "0.9.2.0")]
    internal sealed class ViewsForBaseEntitySets<#= ViewContainerSuffix #> : DbMappingViewCache
    {
        /// <summary>
        /// Gets a hash value computed over the mapping closure.
        /// </summary>
        public override string MappingHashValue
        {
            get { return "<#= MappingHashValue #>"; }
        }

        /// <summary>
        /// Gets a view corresponding to the specified extent.
        /// </summary>
        /// <param name="extent">The extent.</param>
        /// <returns>The mapping view, or null if the extent is not associated with a mapping view.</returns>
        public override DbMappingView GetView(EntitySetBase extent)
        {
            if (extent == null)
            {
                throw new ArgumentNullException("extent");
            }

            var extentName = extent.EntityContainer.Name + "." + extent.Name;
<#
    var index = 0;
    foreach (var view in Views)
    {
#>

            if (extentName == "<#= view.Key.EntityContainer.Name + "." + view.Key.Name #>")
            {
                return GetView<#= index #>();
            }
<#
        index++;
    }
#>

            return null;
        }
<#
    index = 0;
    foreach (var view in Views)
    {
#>

        /// <summary>
        /// Gets the view for <#= view.Key.EntityContainer.Name + "." + view.Key.Name #>.
        /// </summary>
        /// <returns>The mapping view.</returns>
        private static DbMappingView GetView<#= index #>()
        {
            return new DbMappingView(@"<#= view.Value.EntitySql #>");
        }
<#
        index++;
    }
#>
    }
}
<#+
    public string ContextTypeName { get; set; }
 public string ViewContainerSuffix { get; set; }
    public string MappingHashValue { get; set; }
    public dynamic Views { get; set; }
#>

This T4 file is compiled into the class CSharpViewGenerator. Our DbContext can now check the mapping hash value of the loaded precompiled views file in your assembly and compute in again to quickly assert if the database is in sync with the precompiled views. The following code can establish both a check that the precompiled views are in sync and run code that will regenerate the precompiled views file of which the developer can then copy from Notepad and into the precompiled view file again. Not perhaps an elegant solution, but it lets the developer easily check and keep the precompiled views file updated and in sync. And you do not need to install the EF Powertools extension either, as I use the .tt file and much of the source code from there which is part of the "Generate views" command anywyays! Sample DbContext is then:
using System;
using System.Data.Entity;
using System.Data.Entity.Core;
using System.Data.Entity.Core.Mapping;
using System.Data.Entity.Core.Metadata.Edm;
using System.Diagnostics;
using System.IO;
using System.Runtime.CompilerServices;
using Edm_EntityMappingGeneratedViews;

using PrecompiledViewGenerator;

namespace SomeAcme.SomeRegistry.Data
{
    public class SomeDatabaseMigrationsFactory : DatabaseMigrationsFactory<SomeDatabase>
    {
    }

    public class SomeDatabase : SomeDatabaseBaseClass
    {
        public DbSet<SomeEntity> SomeEntity { get; set; }

        public SomeDatabase(ISomeDependency dep)
            : base(dep)
        {
            var someEntityPrecompiledViewsMapping = new ViewsForBaseEntitySetsSomeEntityDatabase();
            var mappingCollection = (StorageMappingItemCollection)ObjectContext.MetadataWorkspace.GetItemCollection(DataSpace.CSSpace);
            string mappingHashValue =  mappingCollection.ComputeMappingHashValue();
            if (SomeEntityPrecompiledViewsMapping.MappingHashValue != mappingHashValue)
            {
                var precompiledViewGenerator = new EntityFrameworkPrecompiledViewGenerator();
                string precompiledViewsContents = precompiledViewGenerator.GeneratePrecompiledViews(this, this.GetType().Name);
                string viewFileForMainDb = "SomeEntityDatabase.Views.cs";
                string temporaryPrecompiledViewFile = Path.Combine(Path.GetTempPath(), viewFileForMainDb);
                try
                {
                    File.WriteAllText(temporaryPrecompiledViewFile, precompiledViewsContents);
                    Process.Start("c:\\windows\\notepad.exe", temporaryPrecompiledViewFile);
                }
                catch (Exception err)
                {
                    throw new Exception("An error occured while trying to regenerate precompiled views for EF since the database changed. Error is: " + err);
                }
                throw new EntityCommandCompilationException($"The precompiled views file is not in sync with the database any longer. Replace the file {viewFileForMainDb} with the generated new contents!");
            }
        }
    }
}

The overall execution of code is the following:
  • In the DbContext constructor - check that the computed mapping hash value matches to that of the of the CSSpace storagemapping collection that your precompiled views file contains (MappingHashValue)
  • If the hash values do not agree, it is necessary to regenerate the pre compiled views file again. The contents are generated and written to a temporary file.
  • Notepad or similar is launched telling the developer to replace the pre compiled views file contents with this new contents.
  • An EntityCommandCompilationException is thrown as this is the same type of exception that is thrown if the precompiled views file does not agree
  • Developer replaces the contents and rebuilds the solution
  • The next time the startup time of EF should be reduced significantly again and work, since our DbContext and precomplied views are in sync again
Below is a sample of an exception thrown when my DbContext was not in sync with the precompiled views file:


System.Data.Entity.Core.EntityCommandCompilationException

System.Data.Entity.Core.EntityCommandCompilationException
  HResult=0x8013193B
  Message=An error occurred while preparing the command definition. See the inner exception for details.
  Source=EntityFramework
  StackTrace:
   at System.Data.Entity.Core.EntityClient.Internal.EntityCommandDefinition..ctor(DbProviderFactory storeProviderFactory, DbCommandTree commandTree, DbInterceptionContext interceptionContext, IDbDependencyResolver resolver, BridgeDataReaderFactory bridgeDataReaderFactory, ColumnMapFactory columnMapFactory)
   at System.Data.Entity.Core.EntityClient.Internal.EntityProviderServices.CreateDbCommandDefinition(DbProviderManifest providerManifest, DbCommandTree commandTree, DbInterceptionContext interceptionContext)
   at System.Data.Entity.Core.Objects.Internal.ObjectQueryExecutionPlanFactory.CreateCommandDefinition(ObjectContext context, DbQueryCommandTree tree)
   at System.Data.Entity.Core.Objects.Internal.ObjectQueryExecutionPlanFactory.Prepare(ObjectContext context, DbQueryCommandTree tree, Type elementType, MergeOption mergeOption, Boolean streaming, Span span, IEnumerable`1 compiledQueryParameters, AliasGenerator aliasGenerator)
   at System.Data.Entity.Core.Objects.ELinq.ELinqQueryState.GetExecutionPlan(Nullable`1 forMergeOption)
   at System.Data.Entity.Core.Objects.ObjectQuery`1.<>c__DisplayClass7.<GetResults>b__6()
   at System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction[T](Func`1 func, IDbExecutionStrategy executionStrategy, Boolean startLocalTransaction, Boolean releaseConnectionOnSuccess)
   at System.Data.Entity.Core.Objects.ObjectQuery`1.<>c__DisplayClass7.<GetResults>b__5()
   at System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.Execute[TResult](Func`1 operation)
   at System.Data.Entity.Core.Objects.ObjectQuery`1.GetResults(Nullable`1 forMergeOption)
   at System.Data.Entity.Core.Objects.ObjectQuery`1.<System.Collections.Generic.IEnumerable<T>.GetEnumerator>b__0()
   at System.Data.Entity.Internal.LazyEnumerator`1.MoveNext()
   at System.Linq.Enumerable.Single[TSource](IEnumerable`1 source)
   atProgram.cs:line 164

Inner Exception 1:
MappingException: The current model no longer matches the model used to pre-generate the mapping views, as indicated by the ViewsForBaseEntitySetsabf8c33ac61e42fc601fe7446b41eaf48c7577efe6d6e17ccccc2b434793c28e.MappingHashValue property. Pre-generated mapping views must be either regenerated using the current model or removed if mapping views generated at runtime should be used instead. See http://go.microsoft.com/fwlink/?LinkId=318050 for more information on Entity Framework mapping views.

 

Saturday 23 February 2019

Serializing a data contract with xml declaration and indented formatting

This code will serialize your object graph and also do xml indentation and adding an xml declaration at the top, using DataContractSerializer.



public static string SerializeObjectIndented(T dataContract, bool omitXmlDeclaration = false) where T : class
{
 using (var output = new StringWriter())
 {
  using (var writer = new XmlTextWriter(output) { Formatting = Formatting.Indented })
  {
   if (!omitXmlDeclaration)
    writer.WriteStartDocument();
   var serializer = new System.Runtime.Serialization.DataContractSerializer(typeof(T));
   serializer.WriteObject(writer, dataContract);
   
   return output.GetStringBuilder().ToString();
  }
 }
}

To use it, just pass in your object and get an xml back!

Tuesday 21 August 2018

Creating a validation attribute for multiple enum values in C#

This article will present a validation attribute for multiple enum value in C#. In C#, generics is not supported in attributes. The following class therefore specifyes the type of enum and provides a list of invalid enum values as an example of such an attribute.


using System;
using System.Collections.Generic;
using System.ComponentModel.DataAnnotations;

namespace ValidateEnums
{

    public sealed class InvalidEnumsAttribute : ValidationAttribute
    {

        private List<object> _invalidValues = new List<object>();



        public InvalidEnumsAttribute(Type enumType, params object[] enumValues)
        {
            foreach (var enumValue in enumValues)
            {
                var _invalidValueParsed = Enum.Parse(enumType, enumValue.ToString());
                _invalidValues.Add(_invalidValueParsed);
            }
        }

        public override bool IsValid(object value)
        {
            foreach (var invalidValue in _invalidValues)
            {
                if (Enum.Equals(invalidValue, value))
                    return false;
            }
            return true;
        }

    }

}

Let us make use of this attribute in a sample class.

 public class Snack
    {
        [InvalidEnums(typeof(IceCream), IceCream.None, IceCream.All )]
        public IceCream IceCream { get; set; }

    }

We can then test out this attribute easily in NUnit tests for example:

[TestFixture]
    public class TestEnumValidationThrowsExpected
    { 

        [Test]
        [ExpectedException(typeof(ValidationException))]
        [TestCase(IceCream.All)]
        [TestCase(IceCream.None)]
        public void InvalidEnumsAttributeTest_ThrowsExpected(IceCream iceCream)
        {
            var snack = new Snack { IceCream = iceCream };
            Validator.ValidateObject(snack, new ValidationContext(snack, null, null), true);
        }

        [Test]
        public void InvalidEnumsAttributeTest_Passes_Accepted()
        {
            var snack = new Snack { IceCream = IceCream.Vanilla };
            Validator.ValidateObject(snack, new ValidationContext(snack, null, null), true);
            Assert.IsTrue(true, "Test passed for valid ice cream!"); 
        }


Wednesday 8 August 2018

Doubly Linked List in C#

I am reading the book "C#7 and .NET Core 2.0 High Performance" about data structures and came accross Doubly linked lists. This is an interesting data structure. We most often use arrays and lists in .NET in everyday use, but both are not as high performant as linked lists when it comes to inserting and removing items in their structure. Removing an item and inserting an item in a list is only quick, if we add to the end of the list or remove from the end of the list. Arrays have the same behavior, if you have a large data structure with many items, consider using a LinkedList instead. .NET already got a good implementation of linked lists in System.Collections.Generic with the LinkedListNode class, so this article just presents a class I wrote for fun on my own. If you want to see the source code of the .Net class, it is available here:
LinkedListNode implementation(Reference Source /.NET)
Now how fun is it to just use .NET's implementation, we want to learn something and do things ourselves as devoted coders? Therefore I present my own implementation! You can find the source code by cloning the following Git repo: Actually, the implementation is easy, the most cumbersome part is to be careful with the next and previous pointers. Just like the .NET implementation, this class supports generic and a payload of different types, in the demo I will use string as the payload of each node.

git clone https://toreaurstad@bitbucket.org/toreaurstad/doublylinkedlist.git


using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
// ReSharper disable ArrangeThisQualifier
// ReSharper disable RedundantNameQualifier

namespace DoublyLinkedList
{

    /// <summary>
    /// Node in a doubly linked list data structure with pointers to the previous and next item in the linked list, if any.
    /// Allows quick insertion and removal of values in even large data structures
    /// </summary>
    /// <typeparam name="T">The type of Data in each node of the doubly linked list</typeparam>
    public class LinkedListNode<T>
    {

        public LinkedListNode(T data)
        {
            _data = data;
            _prev = null; //Init previous reference to null
            _next = null; //Init next reference to null
        }


        public T Data
        {
            get { return _data; }
        }

        /// <summary>
        /// Attempts to find a value in the doubly linked list. Uses object.Equals for comparison. O(N) complexity.
        /// </summary>
        /// <param name="value">Value to find</param>
        /// <returns>The first node with the matching value, if any.</returns>
        public LinkedListNode<T> Find(T value)
        {
            if (object.Equals(Data, value))
                return this;
            if (this._next is null)
                return null;
            return Find(this._next, value);
        }

        /// <summary>
        /// Attempts to find a value in the doubly linked list by a matchen with a given predicate. Returns null if no node values matches. O(N) complexity 
        /// </summary>
        /// <param name="searchCondition"></param>
        /// <returns></returns>
        public LinkedListNode<T> Find(Predicate<LinkedListNode<T>> searchCondition)
        {
            if (searchCondition(this))
                return this;
            if (this._next != null)
                return this._next.Find(searchCondition);
            return null;
        }

        /// <summary>
        /// Searches for multiple values and returns the nodes found. The search returns the first match if any for every search value. O(N*N) complexity.
        /// </summary>
        /// <param name="values"></param>
        /// <returns></returns>
        public LinkedListNode<T>[] FindMultiple(params T[] values)
        {
            if (values is null)
                throw new ArgumentNullException(nameof(values));
            if (!values.Any())
                throw new ArgumentException("Please provide a nonempty array of values!");
            var foundValues = new List<LinkedListNode<T>>();
            foreach (T value in values)
            {
                LinkedListNode<T> foundValue = Find(value);
                if (foundValue != null)
                    foundValues.Add(foundValue);
            }

            return foundValues.ToArray();
        }

        // ReSharper disable once UnusedMember.Local
        private LinkedListNode<T> Find(LinkedListNode<T> node, Predicate<T> searchCondition)
        {
            if (node is null)
                return null;
            if (searchCondition(node.Data))
                return node;
            if (node._next != null)
                return Find(node._next, searchCondition);
            return null;
        }

        private LinkedListNode<T> Find(LinkedListNode<T> node, T value)
        {
            if (node is null)
                return null;
            if (object.Equals(node.Data, value))
                return node;
            if (node._next != null)
                return Find(node._next, value);
            return null;
        }

        /// <summary>
        /// Inserts a node into the doubly linked list. Adjusts the prev and next pointers of the inserted node. O(1) complexity.
        /// </summary>
        /// <param name="node">The node to insert, node's prev and next pointers will be overwritted if already set.</param>
        /// <returns>The inserted node with updated prev and next pointers</returns>
        public LinkedListNode<T> Insert(LinkedListNode<T> node)
        {
            if (node is null)
                throw new ArgumentNullException(nameof(node));

            LinkedListNode<T> nextNode = this._next;
            node._prev = this;
            this._next = node;
            node._next = nextNode;
            if (nextNode != null)
                nextNode._prev = node;

            return node;
        }

        /// <summary>
        /// Inserts multiple nodes into the doubly linked list by building nodes with passed in values. O(1) complexity.
        /// </summary>
        /// <param name="values"></param>
        /// <returns></returns>
        public LinkedListNode<T>[] Insert(params T[] values)
        {
            if (values is null)
                throw new ArgumentNullException(nameof(values));
            if (!values.Any())
                throw new ArgumentException("Please provide a nonempty array of values!");

            values = values.Reverse().ToArray(); //Reverse order so insertion behaves sequentially 

            var inserted = new List<LinkedListNode<T>>();

            foreach (T value in values)
            {
                LinkedListNode<T> node = new LinkedListNode<T>(value);
                inserted.Add(Insert(node));
            }

            return inserted.ToArray();
        }

        /// <summary>
        /// Removes a node from the linked list. Adjusts the previous and next pointers of removed node. O(1) complexity.
        /// </summary>
        /// <param name="node"></param>
        /// <returns></returns>
        public LinkedListNode<T> Remove(LinkedListNode<T> node)
        {
            if (node is null)
                throw new ArgumentNullException(nameof(node)); 

            if (node._prev != null)
                node._prev._next = node._next;
            if (node._next != null)
                node._next._prev = node._prev;

            //Set unneeded references to null now and to avoid misuse
            node._prev = null;
            node._next = null;

            return node;
        }

        public void Remove()
        {
            if (this._prev != null)
                this._prev._next = this._next;
            if (this._next != null)
                this._next._prev = this._prev;

            //Set unneeded references to null now and to avoid misuse
            this._prev = null;
            this._next = null;
        }

        public override string ToString()
        {
            StringBuilder sb = new StringBuilder();
            if (this._prev is null)
                sb.Append(Head);
            sb.Append(_data + GetArrow(this));
            IterateLinkedList(this._next, sb);
            return sb.ToString();
        }

        /// <summary>
        /// Iterates the doubly linked list and builds a string to output in the ToString() method
        /// </summary>
        /// <param name="node">LinkedListNode</param>
        /// <param name="sb">StringBuilder</param>
        private void IterateLinkedList(LinkedListNode<T> node, StringBuilder sb)
        {
            if (node != null)
            {
                sb.Append(node.Data + GetArrow(node));
                if (node._next != null)
                    IterateLinkedList(node._next, sb);
            }
        }

        private string GetArrow(LinkedListNode<T> node)
        {
            if (node != null)
            {
                if (node._next != null && node._next._prev != null)
                    return DoubleArrow;
                if (node._next != null && node._next._prev == null)
                    return Arrow;
                if (node._next == null)
                    return Arrow + NullString;
            }

            return ArrowUndefined;
        }

        private const string Head = "HEAD->";

        private const string Arrow = "->";

        private const string DoubleArrow = "<->";

        private const string ArrowUndefined = "??MISSINGLINK??";

        private const string NullString = "NULL";

        private readonly T _data;

        private LinkedListNode<T> _prev;

        private LinkedListNode<T> _next;

    }
}


This implementation has not got a specific pointer to the head of the list like a circular linked list can provide. The following code makes use of this class to demonstrate its usage:

using System;
using System.Diagnostics;
using System.Linq;

namespace DoublyLinkedList
{
    class Program
    {
        // ReSharper disable once UnusedParameter.Local
        static void Main(string[] args)
        {
            var root = new LinkedListNode("Hello");
            root.Insert(new LinkedListNode("world"));
            root.Insert(new LinkedListNode("Testing"));
            root.Insert(new LinkedListNode("Double linked list!"));

            root.Insert("Inserting", "some", "values!");

            root.Insert("Delete", "me", "please");

            root.FindMultiple("Delete", "me").ToList().ForEach(n => n.Remove(n));

            root.Find(n => n.Data.Contains("pl")).Remove();

            var mismatch = root.Find("Nonexisting value");
            Debug.Assert(mismatch is null, "Expected to not find any item in this doubly linked list with this search value");

            string rootRepresentation = root.ToString();

            Debug.Assert(rootRepresentation == @"HEAD->Hello<->Inserting<->some<->values!<->Double linked list!<->Testing<->world->NULL");

            Console.WriteLine(rootRepresentation);

            Console.ReadKey();
        }
    }
}

The output of this linked list is displaying the contents of the doubly linked list:

HEAD->Hello<->Inserting<->some<->values!<->Double linked list!<->Testing<->world->NULL

As we can see, with our ToString implementation we can deduce that the first and last node is special, the first one lacks a prev pointer illustrated by "HEAD->" and the last node lacks a next pointer illustrated with "->NULL". You will find this in the implementation of the class. I have decided to actually revert the order if the client wants to insert multiple values, as that ordering behaves more naturally. The client can look after a value or multiple values or search with a given predicate, or pass in a node and use it to search. Also, it is possible to remove a node. We end up with an implementation of a Doubly Linked list that can be used in many scenarios. I would advice you to use the .NET version as it supports more features, such as a pointer to the HEAD node. But this implementation is compact and easy to understand for many developers. You will usually use linked list in scenarios where you have much data and want to quickly insert or remove one or several nodes in the linked list. It also supports quick navigation from a node to its previous or next node. Imagine for example working with a class called Book which needs to have an iterable structure ("Pages") to move to the next and previous page and at the same time insert new pages or removing a page from the middle of the data structure. Using an array or a list would be low performant and slow, while a doubly linked list would actually allow the developer to create code that quickly inserts a new page or removes a page at an arbitrary position in the data structure of the Book. This class can of course be optimized to support for example circular linked list with a pointer always to HEAD, or maybe you want to have pointer to HEAD and TAIL and not have a circular list? The source code should be relatively self explanatory for the intermediate C#-developer to revise and improve. I hope you found this article interesting.

Monday 6 August 2018

Simple Base64 encoder in C#

Just read a bit about Base64 encoding and decided to try out writing my own in Linqpad for strings! The way you Base64 encode is to treat each char in the input string as a byte value with groups of six bytes (this byte value is zero padded left) and then mapping the 2^6 values into a Base64 table and outputting the corresponding values into a resulting string. Note that you also pad the Base64 string with '=' char to ensure the entire bit length is divisible with three. That is why Base64 strings in .NET always have 0,1 or 2 '=' chars at the end. The characters used in the Base64 encoding is in .NET the chars [A-z] and [0-9] plus + and slash /.

void Main()
{
 string wordToBase64Encode = "Hello world from base64 encoder. This is a sample input string, does it work?"; 
 string wordBase64EncodedAccordingToNet = Convert.ToBase64String(Encoding.ASCII.GetBytes(wordToBase64Encode)).Dump();
 string wordBase64EncodedAccordingToCustomBase64 = wordToBase64Encode.ToBase64String().Dump();
 (("Are the two strings equal?: ") + string.Equals(wordBase64EncodedAccordingToNet, wordBase64EncodedAccordingToCustomBase64)).Dump();
 
}

public static class LowLevelBase64Extensions {

    private static readonly char[] _base64Table =
 {
  'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 
  'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z',
  'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 
  'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z',
  '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '+', '/',
 };

 public static string ToBase64String(this string toBeEncoded){
     byte[] toBeEncodedByteArray = Encoding.ASCII.GetBytes(toBeEncoded);
  string toBeEncodedBinaryString = 
  string.Join("", 
   toBeEncodedByteArray.Select(b => (Convert.ToString(b, 2).PadLeft(8, '0'))));
   
  //toBeEncodedBinaryString.Length.Dump();
   
  int padCharacters = toBeEncodedBinaryString.Length % 3;
  //toBeEncoded.Dump();
       
  StringWriter sw = new StringWriter();

  for (var i = 0; i < toBeEncodedBinaryString.Length; i = i + 6)
  {
   string encodingToMap =
    toBeEncodedBinaryString.Substring(i, Math.Min(toBeEncodedBinaryString.Length - i, 6))
    .PadRight(6, '0');
   //encodingToMap.Dump();
   byte encodingToMapByte = Convert.ToByte(encodingToMap, 2);
   char encodingMapped = _base64Table[(int)encodingToMapByte];
   sw.Write(encodingMapped);
  }
  
  sw.Write(new String('=', padCharacters));
  
  //Check 24 bits / 3 byte padding 
  return sw.ToString();
 }
}

Note - the casting to int of the encodingToMapByte also works if you cast the byte representation to short or byte. I have compared this custom Base64Encoder with .NET's own Convert.ToBase64String(), and they give equal strings in the use cases I have tried out. Make note that this is just playing around, .NET's own method is optimized for speed. This is more to show what happens with the input string to generate a Base64 encoded string. You can use this code as a basis for custom encoding shemes like Base32. I have used Encoding.ASCII.GetBytes to look up the ASCII code. So this implementation primarily supports input strings in languages using ASCII and Extended ASCII characters, at least that is what I have tested it with. The output in Linqpad shows the following corresponding results:

Monday 23 July 2018

Using MSMQ with WCF and message security

This article will present how you can get started using MSMQ as the communication protocol with WCF and apply message level security. MSMQ has got some strengths compared to HTTP requests and other communication protocols:
  • Requests are default durable, that is if the server is down, the client can send the messages to the server later when it is back again
  • There are no responses as the protocol is one-way and sometimes avoiding a response is a gain
  • All requests are queued and can be made ordered. Using transactional MSMQ queues allows requests only be sent once
  • It can utilize features such as Journal, system queues such as poison messages and dead letters and integrates well with Biztalk
  • It is the most resilient and sturdy communication protocol around and works well also in server-server scenarios and for clients
It also has some weaknesses:
  • There are no responses, so it requires to inspect the queue(s) if something went wrong
  • If there are a lot of messages, chances are that MSMQ queues will go full - it is not the quickest protocol
  • Clients must also have MSMQ installed, not default set up in Windows
  • There is a learning curve for developers and users as MSMQ is less known protocol compared to HTTP, HTTPS and TCP
I have created a sample to get you started with WCF and MSMQ. It will support Message-level security with self signed certificates for client and server. First clone this repo with Git:
git clone https://toreaurstad@bitbucket.org/toreaurstad/demonetmsmqwcfgit.git

Open up the solution in Visual Studio (2017 or 2015). First off, the solution needs to set up self signed certificates for client and server on your developer PC. Run the Script in the Scripts folder in the Host project, the Powershell script is: CreateCertificatesMsmqDemo.ps1 Run it as admin, as it will use Powershell to generate a new certificate, copy over Openssl.exe to c:\openssl folder (if you already have c:\openssl populated, you might want to change this) and convert the certificate to have a RSA format instead of CNG. There are more setup to do, such as selecting your web site in Internet Information Server admin (inetmgr) and setting up enabled protocols to http, net.msmq for the web site for this solution. You need to install MSMQ as a feature in Windows with required subfeatures. Also after the certificates are installed, right click them (in MMC you select Local Computer and Personal certificate store) and select Manage private keys. Now adjust security settings here so that your App Pool user can access the certificates for MSMQ.

Here is where you in Inetmgr (IIS Admin) set up enabled protocols to MSMQ for the web site of this article:
Moving on to the solution itself, I will describe the implementation details. Note that the creation of the MSMQ queue is helped programatically, as the queue cannot be created automatically by WCF. I created a ServiceHostFactory class to create the MSMQ queue that is used in the communication between the client and server.
using System;
using System.Configuration;
using System.Messaging;
using System.ServiceModel;
using System.ServiceModel.Activation;

namespace WcfDemoNetMsmqBinding.Host
{
    public class MessageQueueFactory : ServiceHostFactory
    {

        public override ServiceHostBase CreateServiceHost(string constructorString, Uri[] baseAddresses)
        {
            CreateMsmqIfMissing();
            return base.CreateServiceHost(constructorString, baseAddresses);
        }

        public static void CreateMsmqIfMissing()
        {
            string queueName = string.Format(@".\private$\{0}", MessageQueueName);
            if (!MessageQueue.Exists(queueName))
            {
                MessageQueue createdMessageQueue = MessageQueue.Create(queueName, true);
                string usernName = System.Security.Principal.WindowsIdentity.GetCurrent().Name;
                createdMessageQueue.SetPermissions(usernName, MessageQueueAccessRights.FullControl);

            }
        }

        public static string MessageQueueName
        {
            get
            {
                return ConfigurationManager.AppSettings["QueueName"];

            }
        }

    }
}
This class will create the MessageQueue if it is missing. The QueueName is an appsetting in web config. The .svc file for the Msmq service contains a reference to the factory.

<%@ ServiceHost Language="C#" Debug="true" Service="WcfDemoNetMsmqBinding.Host.MessageQueueService" Factory="WcfDemoNetMsmqBinding.Host.MessageQueueFactory" %>

Now, the setup of the wcf service is done declaratively in web.config (it could be done in code, but I chose to use web.config for most of this sample for defining the MSMQ service). This is the web.config that define the MSMQ service, as you can see it is not very extensive to get started with MSMQ and WCF:

<?xml version="1.0"?>
<configuration>

  <appSettings>
    <add key="aspnet:UseTaskFriendlySynchronizationContext" value="true" />
    <add key="QueueName" value="DemoQueue3" />
  </appSettings>
  <system.web>
    <compilation debug="true" targetFramework="4.6.1" />
    <httpRuntime targetFramework="4.6.1"/>
  </system.web>
  
  <system.serviceModel>

    <services>
      <service name="WcfDemoNetMsmqBinding.Host.MessageQueueService" behaviorConfiguration="NetMsmqBehavior">
        <endpoint contract="WcfDemoNetMsmqBinding.Host.IMessageQueueService" name="NetMsmqEndpoint" 
                  binding="netMsmqBinding" bindingConfiguration="NetMsmq" address="net.msmq://localhost/private/DemoQueue3" />
      </service>    
    </services>

    <bindings>
      <netMsmqBinding>
        <binding name="NetMsmq" durable="true" exactlyOnce="true" receiveErrorHandling="Move" useActiveDirectory="False" queueTransferProtocol="Native">
          <security mode="Message">
            <message clientCredentialType="Certificate"/>
          </security>
        </binding>
      </netMsmqBinding>
    </bindings>
    
      
    <behaviors>
      <serviceBehaviors>
        <behavior name="NetMsmqBehavior">
          <!-- To avoid disclosing metadata information, set the values below to false before deployment -->
          <serviceMetadata httpGetEnabled="true" />
          <!-- To receive exception details in faults for debugging purposes, set the value below to true.  Set to false before deployment to avoid disclosing exception information -->
          <serviceDebug includeExceptionDetailInFaults="true"/>
        <serviceCredentials>
          <serviceCertificate findValue="MSMQWcfDemoserver" storeLocation="LocalMachine" storeName="My" x509FindType="FindBySubjectName" />
            <clientCertificate>
              <certificate findValue="MSMQWcfDemoClient" storeLocation="LocalMachine" storeName="My" x509FindType="FindBySubjectName" />
              <authentication certificateValidationMode="PeerTrust" />
            </clientCertificate>
        </serviceCredentials>
        
        </behavior>
      </serviceBehaviors>
    </behaviors>

  </system.serviceModel>
  <system.webServer>
    <modules runAllManagedModulesForAllRequests="true"/>
    <!--
        To browse web app root directory during debugging, set the value below to true.
        Set to false before deployment to avoid disclosing web app folder information.
      -->
    <directoryBrowse enabled="true"/>
  </system.webServer>

</configuration>

Note that the two certificates that were generated in this sample is set up in serviceCredentials element. We define the serviceCertificae and clientCertificate here. The client takes note to point to the same queue, net.msmq://localhost/private/DemoQueue3 Note that this example is tested with the client and server on same machine. The client could have a local queue in case the server was on a different machine to support durability in the scenario of many computers The client sets up the corresponding certificates in the app.config:
 
 <?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <startup> 
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.6.1" />
    </startup>
    <system.serviceModel>
        <bindings>
            <netMsmqBinding>
                <binding name="NetMsmqEndpoint">
                    <security mode="Message">
                        <message clientCredentialType="Certificate" algorithmSuite="Default" />
                    </security>
                </binding>
            </netMsmqBinding>
        </bindings>
      <behaviors>
        <endpointBehaviors>
          <behavior name="MsmqEndpointBehavior">
            <clientCredentials>
              <clientCertificate storeName="My" storeLocation="LocalMachine" x509FindType="FindBySubjectName"
                                  findValue="MSMQWcfDemoClient" />
              <serviceCertificate>
                <defaultCertificate storeLocation="LocalMachine" storeName="My" x509FindType="FindBySubjectName" findValue="MSMQWcfDemoserver"/>
              </serviceCertificate>
            
            </clientCredentials>
            
          </behavior>
        </endpointBehaviors>
      </behaviors>
        <client>
            <endpoint address="net.msmq://localhost/private/DemoQueue3" binding="netMsmqBinding"
                bindingConfiguration="NetMsmqEndpoint" behaviorConfiguration="MsmqEndpointBehavior" contract="MessageQueueServiceProxy.IMessageQueueService"
                name="NetMsmqEndpoint">
              <identity>
                <dns value="MSMQWcfDemoServer"/>
              </identity>

            </endpoint>
        </client>
    </system.serviceModel>
  <system.web>
    <compilation debug="true" />
  </system.web>
</configuration>

All in all, we end up with a sample where you can communicate between a client and a service using WCF with durable and ordered requests using NetMsmqBinding. As I have presented in earlier articles, the communication is actually in the form of WCF message inside every MSMQ message. The WCF message itself is protected, as this picture shows.

(it is protected using the self signed certificate we generated)
The Powershell script InspectMessageQueueWcfContent.ps1 is included, if you want to inspect the WCF messages themselves of the queue themselves. Note that if you enable Journal on the MSMQ queue this solution creates, you can see the MSMQ messages after they have been received and consumed, using compmgmt.msc Or an alternative is to use QueueExplorer instead, available as a trial here: Queue Explorer
This tool can view the WCF message inside the MSMQ message as in my Powershell script, but also display syntax coloring and other functionality for quickly navigation of your MSMQ queues. Are MSMQ in WCF an alternative for ordinary scenarios using HTTP, TCP or Federated bindings? This article was meant at least to give a demo of the capability WCF gives developers to utilize MSMQ as the communication protocol between client and server. Hope you found this article interesting.