Saturday 18 May 2024

Discriminated Union Part Two - The C# side of things

In this article , discriminated unions will be further looked into, continuing from the last article. It visited these topics using F#. The previous article showing the previous article focused on F# and discriminated unions is available here:

https://toreaurstad.blogspot.com/2024/05/discriminated-unions-part-one-f-side-of.html

In this article, C# will be used. As last noted, discriminated unions are a set of types that are allowed to be used. In F#, these types dont have to be in an inheritance chain, they can really be a mix of different types. In C# however, one has to use a base type for the union itself and declare this as abstract, i.e. a placeholder for our discriminated union, called DU from now in this article. C# is a mix of object oriented and functional programming language. It does not support discriminated unions as built-in constructs, such as F#. We must use object inheritance still, but pattern matching in C# with type testing. Lets first look at the POCOs that are included in this example, we must use a base class for our union. In F# we had this:


type Shape =
    | Rectangle of width : float * length : float
    | Circle of radius : float
    | Prism of width : float * depth:float * height : float
    | Cube of width : float


In C# we use an abstract record, since they possess immutability after construction has been made and are therefore a good match for functional programming (FP). Also, records offer a compact syntax which lets itself nice to FP. (we COULD use an abstract class too, but records are now available and lends themsevles better to FP since they are immutable after construction is finished). We could define this abstract record baseclass which will function as a Discriminated Union (DU) like:


public abstract record Shape;


However, keeping note of which types are allowed into the DU is easier if we nest our types. I have also included the methods on the Shape objects as static methods that uses pattern matching with type testing to define the bounds of our DU.



public abstract record Shape
{

	public record Rectangle(float Width, float Length) : Shape;
	public record Circle(float Radius) : Shape;
	public record Prism(float Width, float Depth, float Length) : Shape;
	public record Cube(float Width) : Shape;
	public record Torus(float LargeRadius, float SmallRadius) : Shape; //we will discriminate this shape, not include it in our supported calculations

	public static double CalcArea(Shape shape) => shape switch
	{
		Rectangle rect => rect.Width * rect.Length,
		Circle circ => Math.PI * Math.Pow(circ.Radius, 2),
		Prism prism => 2.0*(prism.Width*prism.Depth) + 2.0*(prism.Width+prism.Depth)*prism.Length,
		Cube cube => 6 * Math.Pow(cube.Width, 2),
		_ => throw new NotSupportedException($"Area calculation for this Shape: ${shape.GetType()}")
	};

	public static double CalcVolume(Shape shape) => shape switch
	{
		Prism prism => prism.Width * prism.Depth * prism.Length,
		Cube cube => Math.Pow(cube.Width, 3),
		_ => throw new NotSupportedException($"Volume calculation for this Shape: ${shape.GetType()}")
	};

};


Sample code of using this source code is shown below:


void Main()
{
	var torus = new Shape.Torus(LargeRadius: 7, SmallRadius: 3);
	//var torusArea = Shape.CalcArea(torus);

	var rect = new Shape.Rectangle(Width: 1.3f, Length: 10.0f);
	var circle = new Shape.Circle(Radius: 2.0f);
	var prism = new Shape.Prism(Width: 15, Depth: 5, Length: 7);
	var cube = new Shape.Cube(Width: 2.0f);

	var rectArea = Shape.CalcArea(rect);
	var circleArea = Shape.CalcArea(circle);
	var prismArea = Shape.CalcArea(prism);
	var cubeArea = Shape.CalcArea(cube);

	//var circleVolume = Shape.CalcVolume(circle);
	var prismVolume = Shape.CalcVolume(prism);
	var cubeVolume = Shape.CalcVolume(cube);
	//var rectVolume = Shape.CalcVolume(rect);

	Console.WriteLine("\nAREA CALCULATIONS:");
	Console.WriteLine($"Circle area: {circleArea:F2}");
	Console.WriteLine($"Prism area: {prismArea:F2}");
	Console.WriteLine($"Cube area: {cubeArea:F2}");
	Console.WriteLine($"Rectangle area: {rectArea:F2}");

	Console.WriteLine("\nVOLUME CALCULATIONS:");
	//Console.WriteLine( "Circle volume: %A", circleVolume);
	Console.WriteLine($"Prism volume: {prismVolume:F2}");
	Console.WriteLine($"Cube volume: {cubeVolume:F2}");
	//Console.WriteLine( "Rectangle volume: %A", rectVolume);
}


I have commented out some lines here, they will throw an UnsupportedException if one uncomments them running the code. The torus forexample lacks support for area and volume calculation by intent, it is not supported (yet). The calculations of the volume of a circle and a rectangle is not possible, since they are 2D geometric figures and not 3D, i.e. do not posess a volume. Output from running the program is shown below:

AREA CALCULATIONS:
Circle area: 12,57
Prism area: 430,00
Cube area: 24,00
Rectangle area: 13,00

VOLUME CALCULATIONS:
Prism volume: 525,00
Cube volume: 8,00

Conclusions F# vs C#

True support for DU is only available in F#, but we can get close to it using C#, inheritance, pattern matching with type checking. F# got much better support for it for now, but C# probably will catch up in a few years and also finally get support for it as a built-in construct. The syntax for DU in F# an C# is fairly similar, using records and pattern switching with type checking makes the code in C# not longer than in F#, but F# got direct support for DU, in C# we have to add additional code to support something that is a built-in functionality of F#. Listed on the page What's new in C# 13, DU has not made their way into the list, .NET 9 Preview SDK will be available probably in November this year (2024).

https://learn.microsoft.com/en-us/dotnet/csharp/whats-new/csharp-13

There are different approaches to writing DU in C# for now. Some go for the OneOf operator of functional programming, not presented further in this article. Probably discriminated unions will make their way in .NET 10 in November 2027, so there will still be a lot of waiting around for getting this feature into C#. For now, being aware what the buzz about DU is all about, my two articles on it hopefully made it a bit clearer. One disadvantage of this is that it’s not consistent like in F#. We have to manually manage which types we want to support in each method. However, this is done using inheritance in C#. At the same time, we need to adjust the inheritance hierarchy so that all types inherit from such a discriminated union (DU). If a type needs to be part of MULTIPLE different DUs, we face limitations in C# since we can only inherit from a specific type in the hierarchy. This is likely why many C# developers are requesting DU functionality. As of now, Microsoft’s language team seems to be leaning toward something called ENUM CLASSES. It appears that this feature will be included in .NET 10, which means it won’t be available until 2027

Further viewing/reading of the topic

There are proposals for better support of DU in C# is taking its form now in concrete propals. A proposal for Enum classes are available here, it could be the design choice C# language team lands on:

https://github.com/dotnet/csharplang/blob/main/proposals/discriminated-unions.md

Lead Designer Mads Torgersen comments around DU in C# in this video at 21:00 :

https://learn.microsoft.com/en-us/shows/ask-the-expert/ask-the-expert-whats-new-in-c-100

Saturday 11 May 2024

Discriminated Unions Part One - The F# side of things

I decided to look more into what the discussion of Discriminated unions in C#, or their lack of it is all about. I will first look at the F# side of things. How can we create a discriminated union in F# ? And then I will look at how we can implement the F# program in C# in the next article for the topic Discriminated unions. In this article we will look at some F# code that shows how discriminated unions are built-in supported in F#. Discriminated unions are special containers that can hold different types. This is not supported in C# without adding some additional plumbing code and it is not considered true discriminated unions, although in C# we can get close to Discriminated unions. For the rest of the article, we will call discriminated unions for DU. Let's first declare a DU in F# that describes different types of geometric figures.


type Shape =
    | Rectangle of width : float * length : float
    | Circle of radius : float
    | Prism of width : float * depth:float * height : float
    | Cube of width : float


The '*' operator in F# means when it is used in type definitions above as a separator of the properties that each type got,
e.g. Rectangle of width : float * length : float means
that the type Rectangle got two properties, width of type float and length of the same type.

Let's add some methods to our F# program, calculating area and calculating volume. We also want our F# to be fault tolerant so either we get a result or we get an error, for example this additional DU which is also generic.

type Result<'T> =
    | Success of 'T
    | Error of string    

We also neeed a way to print errors if we want to not crash the program, say if want to calculate the volume of a circle or a rectangle, which is not supported since it is 2D figures.


let handleResult (result: Result<float>) =
    match result with
    | Success value -> printfn "%A" value
    | Error msg -> printfn "Error: %s" msg; () // Return NaN for error cases


To add some functionality to the discriminated unions we add the module below:


module ShapeOperations =
    let CalcArea(shape : Shape) : Result<float> =
        match shape with 
        | Rectangle (width, length) -> Success(width * length)
        | Circle (radius) -> Success(Math.PI * radius**2)
        | Prism (width, depth, height) -> (2.0*(width*depth) + 2.0*(width+depth)*height)
        | Cube (width) -> Success(6.0 * width * width)
        // | _ -> failwith "Area calculation is not supported"
    let CalcVolume(shape : Shape) : Result<float> = 
        match shape with 
        | Prism (width, height, depth) -> Success(width * height * depth)
        | Cube (width) -> Success(width**3)        
        | _ -> Error(sprintf "Volume calculation is not supported  for: %A" shape)


The rest of the code is shown below where we instantiate geometric figures and calculate the area and volume of them and output their values.

 
let rect = Rectangle(length = 1.3, width = 10.0)
let circle = Circle (2.0)
let prism = Prism(width = 15, depth = 5.0, height = 7.0)
let cube = Cube(3)

let rectArea = ShapeOperations.CalcArea rect 
let circleArea = ShapeOperations.CalcArea circle
let prismArea = ShapeOperations.CalcArea prism
let cubeArea = ShapeOperations.CalcArea cube

let circleVolume = handleResult (ShapeOperations.CalcVolume circle)
let prismVolume = ShapeOperations.CalcVolume prism
let cubeVolume = ShapeOperations.CalcVolume cube
let rectVolume = ShapeOperations.CalcVolume rect

printfn "\nAREA CALCULATIONS:"
printfn "Circle area: %A" circleArea
printfn "Prism area: %A" prismArea 
printfn "Cube area: %A" cubeArea 
printfn "Rectangle area %A" rectArea 

printfn "\nVOLUME CALCULATIONS:"
printfn "Circle volume: %A" circleVolume 
printfn "Prism volume: %A" prismVolume 
printfn "Cube volume: %A" cubeVolume 
printfn "Rectangle volume: %A" rectVolume          
                      

We get this output after running the program :


Error: Volume calculation is not supported  for: Circle 2.0

AREA CALCULATIONS:
Circle area: Success 12.56637061
Prism area: Success 430
Cube area: Success 18.0
Rectangle area Success 13.0

VOLUME CALCULATIONS:
Circle volume: ()
Prism volume: Success 525.0
Cube volume: Success 27.0
Rectangle volume: Error "Volume calculation is not supported  for: Rectangle (10.0, 1.3)"


As we can see, creating DUs in F# is easy, we use the '|' operator to define multiple types and we can create generic DUs too and match different types with functional expressions. In the next article we will look at the code shown here and test out if we can recreate it in C# using different constructs. C# has gotten more support of functional programming in 2020 and most likely it will involve records, pattern matching (newer switch based syntax) and extension methods.

Thursday 9 May 2024

Azure Cognitive Synthesized Text To Speech with voice styles

Using Azure Cognitive Services, it is possible to translate text into other languages and also synthesize the text to speech. It is also possible to add voice effects such as style of the voice. This adds more realism by adding emotions to a synthesized voice. The voice is already trained by neural net training and adding voice style makes the synthesized speech even more realistic and multi-purpose. The Github repo for this is available here as .NET Maui Blazor client written with .NET 8 :

MultiLingual translator DEMO Github repo

Not all the voices supported in Azure Cognitive Services do support voice effects. An overview of which voices are shown here:

https://learn.microsoft.com/nb-no/azure/ai-services/speech-service/language-support?tabs=tts#voice-styles-and-roles

More and more synthetic voices in Azure Cognitive Services gets more and more voice styles which express emotions. For now, most of the voices are either english (en-US) or chinese (zh-CN) and a few other languages got some few voices supporting styles. This will most likely be improved into the future where these neural net trained voices are trained in voice styles or some generic voice style algorithm is achieved that can infer emotions on a generic level, although that still sounds a bit sci-fi.

Azure Cognitive Text-To-Speech Voices with support for emotions / voice styles


Voice Styles Roles
de-DE-ConradNeural1 cheerful Not supported
en-GB-SoniaNeural cheerful, sad Not supported
en-US-AriaNeural angry, chat, cheerful, customerservice, empathetic, excited, friendly, hopeful, narration-professional, newscast-casual, newscast-formal, sad, shouting, terrified, unfriendly, whispering Not supported
en-US-DavisNeural angry, chat, cheerful, excited, friendly, hopeful, sad, shouting, terrified, unfriendly, whispering Not supported
en-US-GuyNeural angry, cheerful, excited, friendly, hopeful, newscast, sad, shouting, terrified, unfriendly, whispering Not supported
en-US-JaneNeural angry, cheerful, excited, friendly, hopeful, sad, shouting, terrified, unfriendly, whispering Not supported
en-US-JasonNeural angry, cheerful, excited, friendly, hopeful, sad, shouting, terrified, unfriendly, whispering Not supported
en-US-JennyNeural angry, assistant, chat, cheerful, customerservice, excited, friendly, hopeful, newscast, sad, shouting, terrified, unfriendly, whispering Not supported
en-US-NancyNeural angry, cheerful, excited, friendly, hopeful, sad, shouting, terrified, unfriendly, whispering Not supported
en-US-SaraNeural angry, cheerful, excited, friendly, hopeful, sad, shouting, terrified, unfriendly, whispering Not supported
en-US-TonyNeural angry, cheerful, excited, friendly, hopeful, sad, shouting, terrified, unfriendly, whispering Not supported
es-MX-JorgeNeural chat, cheerful Not supported
fr-FR-DeniseNeural cheerful, sad Not supported
fr-FR-HenriNeural cheerful, sad Not supported
it-IT-IsabellaNeural chat, cheerful Not supported
ja-JP-NanamiNeural chat, cheerful, customerservice Not supported
pt-BR-FranciscaNeural calm Not supported
zh-CN-XiaohanNeural affectionate, angry, calm, cheerful, disgruntled, embarrassed, fearful, gentle, sad, serious Not supported
zh-CN-XiaomengNeural chat Not supported
zh-CN-XiaomoNeural affectionate, angry, calm, cheerful, depressed, disgruntled, embarrassed, envious, fearful, gentle, sad, serious Boy, Girl, OlderAdultFemale, OlderAdultMale, SeniorFemale, SeniorMale, YoungAdultFemale, YoungAdultMale
zh-CN-XiaoruiNeural angry, calm, fearful, sad Not supported
zh-CN-XiaoshuangNeural chat Not supported
zh-CN-XiaoxiaoNeural affectionate, angry, assistant, calm, chat, chat-casual, cheerful, customerservice, disgruntled, fearful, friendly, gentle, lyrical, newscast, poetry-reading, sad, serious, sorry, whisper Not supported
zh-CN-XiaoyiNeural affectionate, angry, cheerful, disgruntled, embarrassed, fearful, gentle, sad, serious Not supported
zh-CN-XiaozhenNeural angry, cheerful, disgruntled, fearful, sad, serious Not supported
zh-CN-YunfengNeural angry, cheerful, depressed, disgruntled, fearful, sad, serious Not supported
zh-CN-YunhaoNeural2 advertisement-upbeat Not supported
zh-CN-YunjianNeural3,4 angry, cheerful, depressed, disgruntled, documentary-narration, narration-relaxed, sad, serious, sports-commentary, sports-commentary-excited Not supported
zh-CN-YunxiaNeural angry, calm, cheerful, fearful, sad Not supported
zh-CN-YunxiNeural angry, assistant, chat, cheerful, depressed, disgruntled, embarrassed, fearful, narration-relaxed, newscast, sad, serious Boy, Narrator, YoungAdultMale
zh-CN-YunyangNeural customerservice, narration-professional, newscast-casual Not supported
zh-CN-YunyeNeural angry, calm, cheerful, disgruntled, embarrassed, fearful, sad, serious Boy, Girl, OlderAdultFemale, OlderAdultMale, SeniorFemale, SeniorMale, YoungAdultFemale, YoungAdultMale
zh-CN-YunzeNeural angry, calm, cheerful, depressed, disgruntled, documentary-narration, fearful, sad, serious OlderAdultMale, SeniorMale

Screenshot from the DEMO showing its user interface. You enter the text to translate at the top and the language of the text is detected using Azure Cognitive Services text detection functionality. And you can then select which language to translate the text into. It will call a REST call to Azure Cognitive Services to translate the text. And it is also possible to hear the speech of the text. Now, it is also added to add voice style. Use the table shown above to select a voice actor that supports a voice style you want to test. As noted, voice styles are still limited to a few languages and voice actors supporting emotions or voice styles. You will hear the voice from the voice actor in a normal mood or voice style if additional emotions or voice styles are not supported.
Let's look at some code for this DEMO too. You can study the Github repo and clone it to test it out yourself. The TextToSpeechUtil class handles much of the logic of creating voice from text input and also create the SSML-XML contents and performt the REST api call to create the voice file. Note that SSML mentioned here, is the Speech Synthesis Markup Language (SSML). The SSML standard is documented here on MSDN, it is a standard adopted by others too including Google.

https://learn.microsoft.com/en-us/azure/ai-services/speech-service/speech-synthesis-markup



using Microsoft.Extensions.Configuration;
using MultiLingual.Translator.Lib.Models;
using System;
using System.Security;
using System.Text;
using System.Xml.Linq;
using static System.Runtime.InteropServices.JavaScript.JSType;

namespace MultiLingual.Translator.Lib
{
    public class TextToSpeechUtil : ITextToSpeechUtil
    {

        public TextToSpeechUtil(IConfiguration configuration)
        {
            _configuration = configuration;
        }

        public async Task<TextToSpeechResult> GetSpeechFromText(string text, string language, TextToSpeechLanguage[] actorVoices, 
            string? preferredVoiceActorId, string? preferredVoiceStyle)
        {
            var result = new TextToSpeechResult();

            result.Transcript = GetSpeechTextXml(text, language, actorVoices, preferredVoiceActorId, preferredVoiceStyle, result);
            result.ContentType = _configuration[TextToSpeechSpeechContentType];
            result.OutputFormat = _configuration[TextToSpeechSpeechXMicrosoftOutputFormat];
            result.UserAgent = _configuration[TextToSpeechSpeechUserAgent];
            result.AvailableVoiceActorIds = ResolveAvailableActorVoiceIds(language, actorVoices);
            result.LanguageCode = language;

            string? token = await GetUpdatedToken();

            HttpClient httpClient = GetTextToSpeechWebClient(token);

            string ttsEndpointUrl = _configuration[TextToSpeechSpeechEndpoint];
            var response = await httpClient.PostAsync(ttsEndpointUrl, new StringContent(result.Transcript, Encoding.UTF8, result.ContentType));

            using (var memStream = new MemoryStream()) {
                var responseStream = await response.Content.ReadAsStreamAsync();
                responseStream.CopyTo(memStream);
                result.VoiceData = memStream.ToArray();
            }

            return result;
        }

        private async Task<string?> GetUpdatedToken()
        {
            string? token = _token?.ToNormalString();
            if (_lastTimeTokenFetched == null || DateTime.Now.Subtract(_lastTimeTokenFetched.Value).Minutes > 8)
            {
                token = await GetIssuedToken();
            }

            return token;
        }

        private HttpClient GetTextToSpeechWebClient(string? token)
        {
            var httpClient = new HttpClient();
            httpClient.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", token);
            httpClient.DefaultRequestHeaders.Add("X-Microsoft-OutputFormat", _configuration[TextToSpeechSpeechXMicrosoftOutputFormat]);
            httpClient.DefaultRequestHeaders.Add("User-Agent", _configuration[TextToSpeechSpeechUserAgent]);
            return httpClient;
        }
       
        public string GetSpeechTextXml(string text, string language, TextToSpeechLanguage[] actorVoices, string? preferredVoiceActorId,
              string? preferredVoiceStyle, TextToSpeechResult result)
        {
            result.VoiceActorId = ResolveVoiceActorId(language, preferredVoiceActorId, actorVoices);
            string speechXml = $@"
            <speak version='1.0' xml:lang='en-US' xmlns:mstts='https://www.w3.org/2001/mstts'>
                <voice xml:gender='Male' name='Microsoft Server Speech Text to Speech Voice {result.VoiceActorId}'>
                    <prosody rate='1'>{text}</prosody>
                </voice>
            </speak>";

            speechXml = AddVoiceStyleEffectIfDesired(preferredVoiceStyle, speechXml);

            return speechXml;
        }

        /// <summary>
        /// Adds voice style / expression to the SSML markup for the voice
        /// </summary>
        private static string AddVoiceStyleEffectIfDesired(string? preferredVoiceStyle, string speechXml)
        {
            if (!string.IsNullOrWhiteSpace(preferredVoiceStyle) && preferredVoiceStyle != "normal-neutral")
            {
                var voiceDoc = XDocument.Parse(speechXml); //https://learn.microsoft.com/nb-no/azure/ai-services/speech-service/speech-synthesis-markup-voice#use-speaking-styles-and-roles

                XElement? prosody = voiceDoc.Descendants("prosody").FirstOrDefault();
                if (prosody?.Value != null)
                {
                    // Create the <mstts:express-as> element, for now skip the ':' letter and replace at the end

                    var expressedAsWrappedElement = new XElement("msttsexpress-as",
                        new XAttribute("style", preferredVoiceStyle));
                    expressedAsWrappedElement.Value = prosody!.Value;
                    prosody?.ReplaceWith(expressedAsWrappedElement);
                    speechXml = voiceDoc.ToString().Replace(@"msttsexpress-as", "mstts:express-as");
                }
            }

            return speechXml;
        }

        private List<string> ResolveAvailableActorVoiceIds(string language, TextToSpeechLanguage[] actorVoices)
        {
            if (actorVoices?.Any() == true)
            {
                var voiceActorIds = actorVoices.Where(v => v.LanguageKey == language || v.LanguageKey.Split("-")[0] == language).SelectMany(v => v.VoiceActors).Select(v => v.VoiceId).ToList();
                return voiceActorIds;
            }
            return new List<string>();
        }

        private string ResolveVoiceActorId(string language, string? preferredVoiceActorId, TextToSpeechLanguage[] actorVoices)
        {
            string actorVoiceId = "(en-AU, NatashaNeural)"; //default to a select voice actor id 
            if (actorVoices?.Any() == true)
            {
                var voiceActorsForLanguage = actorVoices.Where(v => v.LanguageKey == language || v.LanguageKey.Split("-")[0] == language).SelectMany(v => v.VoiceActors).Select(v => v.VoiceId).ToList();
                if (voiceActorsForLanguage != null)
                {
                    if (voiceActorsForLanguage.Any() == true)
                    {
                        var resolvedPreferredVoiceActorId = voiceActorsForLanguage.FirstOrDefault(v => v == preferredVoiceActorId);
                        if (!string.IsNullOrWhiteSpace(resolvedPreferredVoiceActorId))
                        {
                            return resolvedPreferredVoiceActorId!;
                        }
                        actorVoiceId = voiceActorsForLanguage.First();
                    }
                }
            }
            return actorVoiceId;
        }

        private async Task<string> GetIssuedToken()
        {
            var httpClient = new HttpClient();
            string? textToSpeechSubscriptionKey = Environment.GetEnvironmentVariable("AZURE_TEXT_SPEECH_SUBSCRIPTION_KEY", EnvironmentVariableTarget.Machine);
            httpClient.DefaultRequestHeaders.Add(OcpApiSubscriptionKeyHeaderName, textToSpeechSubscriptionKey);
            string tokenEndpointUrl = _configuration[TextToSpeechIssueTokenEndpoint];
            var response = await httpClient.PostAsync(tokenEndpointUrl, new StringContent("{}"));
            _token = (await response.Content.ReadAsStringAsync()).ToSecureString();
            _lastTimeTokenFetched = DateTime.Now;
            return _token.ToNormalString();
        }

        public async Task<List<string>> GetVoiceStyles()
        {
            var voiceStyles = new List<string>
            {
                "normal-neutral",
                "advertisement_upbeat",
                "affectionate",
                "angry",
                "assistant",
                "calm",
                "chat",
                "cheerful",
                "customerservice",
                "depressed",
                "disgruntled",
                "documentary-narration",
                "embarrassed",
                "empathetic",
                "envious",
                "excited",
                "fearful",
                "friendly",
                "gentle",
                "hopeful",
                "lyrical",
                "narration-professional",
                "narration-relaxed",
                "newscast",
                "newscast-casual",
                "newscast-formal",
                "poetry-reading",
                "sad",
                "serious",
                "shouting",
                "sports_commentary",
                "sports_commentary_excited",
                "whispering",
                "terrified",
                "unfriendly"
            };
            return await Task.FromResult(voiceStyles);
        }

        private const string OcpApiSubscriptionKeyHeaderName = "Ocp-Apim-Subscription-Key";
        private const string TextToSpeechIssueTokenEndpoint = "TextToSpeechIssueTokenEndpoint";
        private const string TextToSpeechSpeechEndpoint = "TextToSpeechSpeechEndpoint";        
        private const string TextToSpeechSpeechContentType = "TextToSpeechSpeechContentType";
        private const string TextToSpeechSpeechUserAgent = "TextToSpeechSpeechUserAgent";
        private const string TextToSpeechSpeechXMicrosoftOutputFormat = "TextToSpeechSpeechXMicrosoftOutputFormat";

        private readonly IConfiguration _configuration;

        private DateTime? _lastTimeTokenFetched = null;
        private SecureString _token = null;

    }
}

 
 

The REST call to generate the voice file is using following set up: TTS endpoint url: https://norwayeast.tts.speech.microsoft.com/cognitiveservices/v1 The transcript (text to translate into speech) is the following in my test as a SSML-XML document:


<speak version="1.0" xml:lang="en-US" xmlns:mstts="https://www.w3.org/2001/mstts">
  <voice xml:gender="Male" name="Microsoft Server Speech Text to Speech Voice (en-US, JaneNeural)">
    <mstts:express-as style="angry">I listen to Eurovision and cheer for Norway</mstts:express-as>
  </voice>
</speak>


The SSML also contains an extension called mstts extension language that adds features to SSML such as the express-as set to a voice style or emotion of "angry". Not all emotions or voice styles are supported by every voice actor in Azure Cognitive Services. But this is a list of the voice styles that could be supported, it varies which voice actor you choose (and inherently which language).
  • "normal-neutral"
  • "advertisement_upbeat"
  • "affectionate"
  • "angry"
  • "assistant"
  • "calm"
  • "chat"
  • "cheerful"
  • "customerservice"
  • "depressed"
  • "disgruntled"
  • "documentary-narration"
  • "embarrassed"
  • "empathetic"
  • "envious"
  • "excited"
  • "fearful"
  • "friendly"
  • "gentle"
  • "hopeful"
  • "lyrical"
  • "narration-professional"
  • "narration-relaxed"
  • "newscast"
  • "newscast-casual"
  • "newscast-formal"
  • "poetry-reading"
  • "sad"
  • "serious"
  • "shouting"
  • "sports_commentary"
  • "sports_commentary_excited"
  • "whispering"
  • "terrified"
  • "unfriendly
Microsoft has come a long way from the early work with SAPI - Microsoft Speech API with Microsoft SAM around 2000. The realism of synthetic voices more than 20 years ago were rather crude and robotic. Nowaydays, voice actors provided by Azure Cloud computing platform as shown here are neural net trained and very realistic based upon training from real voice actors and now more and more voice actor voices support emotions or voice styles. The usages of this can be diverse. Making use of text synthesis can serve in automated answering services and apps in diverse fields such as healthcare and public services or education and more. Making this demo has been fun for me and it can be used to learn languages and with the voice functionality you can train on not only the translation but also pronounciation.

Monday 22 April 2024

Pii - Detecting Personally Identifiable Information using Azure Cognitive Services

This article will look at detecting Person Identifiable Information (Pii) using Azure Cognitive Services. I have created a demo using .NET Maui Blazor has been created and the Github repo is here:
https://github.com/toreaurstadboss/PiiDetectionDemo

Person Identifiable Information (Pii) is desired to detect and also redact, that is using censorship or obscuring Pii to prepare documents for publication. The Pii feature in Azure Cognitive Services is a part of the Language resource service. A quickstart for using Pii is available here:
https://learn.microsoft.com/en-us/azure/ai-services/language-service/personally-identifiable-information/quickstart?pivots=programming-language-csharp

After creating the Language resource, look up the keys and endpoints for you service. Using Azure CLI inside Cloud shell, you can enter this command to find the keys, in Azure many services has got two keys you can exchange with new keys through regeneration:

az cognitiveservices account keys list --resource-group SomeAzureResourceGroup --name SomeAccountAzureCognitiveServices
This is how you can query after endpoint of language resource using Azure CLI : az cognitiveservices account show --query "properties.endpoint" --resource-group SomeAzureResourceGroup --name SomeAccountAzureCognitiveServices
Next, the demo of this article. Connecting to the Pii Removal Text Analytics is possible using this Nuget package (REST calls can also be done manually): - Azure.AI.TextAnalytics version 5.3.0 Here is the other Nugets of my Demo included from the .csproj file :

PiiDetectionDemo.csproj


  <ItemGroup>
        <PackageReference Include="Azure.AI.TextAnalytics" Version="5.3.0" />
        <PackageReference Include="Microsoft.Maui.Controls" Version="$(MauiVersion)" />
        <PackageReference Include="Microsoft.Maui.Controls.Compatibility" Version="$(MauiVersion)" />
        <PackageReference Include="Microsoft.AspNetCore.Components.WebView.Maui" Version="$(MauiVersion)" />
        <PackageReference Include="Microsoft.Extensions.Logging.Debug" Version="8.0.0" />
    </ItemGroup>


A service using this Pii removal feature is simply making use of a TextAnalyticsClient and method RecognizePiiEntitiesAsync.

PiiRemovalTextClientService.cs IPiiRemovalTextClientService.cs



using Azure;
using Azure.AI.TextAnalytics;

namespace PiiDetectionDemo.Util
{
    public interface IPiiRemovalTextAnalyticsClientService
    {
        Task<Response<PiiEntityCollection>> RecognizePiiEntitiesAsync(string? document, string? language);
    }
}


namespace PiiDetectionDemo.Util
{
    public class PiiRemovalTextAnalyticsClientService : IPiiRemovalTextAnalyticsClientService
    {

        private TextAnalyticsClient _client;

        public PiiRemovalTextAnalyticsClientService()
        {
            var azureEndpoint = Environment.GetEnvironmentVariable("AZURE_COGNITIVE_SERVICE_ENDPOINT");
            var azureKey = Environment.GetEnvironmentVariable("AZURE_COGNITIVE_SERVICE_KEY");

            if (string.IsNullOrWhiteSpace(azureEndpoint))
            {
                throw new ArgumentNullException(nameof(azureEndpoint), "Missing system environment variable: AZURE_COGNITIVE_SERVICE_ENDPOINT");
            }
            if (string.IsNullOrWhiteSpace(azureKey))
            {
                throw new ArgumentNullException(nameof(azureKey), "Missing system environment variable: AZURE_COGNITIVE_SERVICE_KEY");
            }

            _client = new TextAnalyticsClient(new Uri(azureEndpoint), new AzureKeyCredential(azureKey));
        }

        public async Task<Response<PiiEntityCollection>> RecognizePiiEntitiesAsync(string? document, string? language)
        {
            var piiEntities = await _client.RecognizePiiEntitiesAsync(document, language);
            return piiEntities;
        }

    }
}


The UI codebehind of the razor component page showing the UI looks like this:

Home.razor.cs


using Azure;
using Microsoft.AspNetCore.Components;
using PiiDetectionDemo.Models;
using PiiDetectionDemo.Util;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace PiiDetectionDemo.Components.Pages
{
    public partial class Home
    {

        private IndexModel Model = new();
        private bool isProcessing = false;
        private bool isSearchPerformed = false;

        private async Task Submit()
        {
            isSearchPerformed = false;
            isProcessing = true;
            try
            {
                var response = await _piiRemovalTextAnalyticsClientService.RecognizePiiEntitiesAsync(Model.InputText, null);
                Model.RedactedText = response?.Value?.RedactedText;
                Model.UpdateHtmlRedactedText();
                Model.AnalysisResult = response?.Value;
                StateHasChanged();
            }
            catch (Exception ex)
            {
                await Console.Out.WriteLineAsync(ex.ToString());
            }
            isProcessing = false;
            isSearchPerformed = true;
        }

        private void removeWhitespace(ChangeEventArgs args)
        {
            Model.InputText = args.Value?.ToString()?.CleanupAllWhiteSpace();
            StateHasChanged();
        }



    }
}



To get the redacted or censored text void of any Pii that the Pii detection feature was able to detect, access the Value of type Azure.AI.TextAnalytics.PiiEntityCollection. Inside this object, the string RedactedText contains the censored / redacted text. The IndexModel looks like this :


using Azure.AI.TextAnalytics;
using Microsoft.AspNetCore.Components;
using PiiDetectionDemo.Util;
using System.ComponentModel.DataAnnotations;
using System.Text;

namespace PiiDetectionDemo.Models
{

    public class IndexModel
    {

        [Required]
        public string? InputText { get; set; }

        public string? RedactedText { get; set; }

        public string? HtmlRedactedText { get; set; }

        public MarkupString HtmlRedactedTextMarkupString { get; set; }

        public void UpdateHtmlRedactedText()
        {
            var sb = new StringBuilder(RedactedText);
            if (AnalysisResult != null && RedactedText != null)
            {
                foreach (var piiEntity in AnalysisResult.OrderByDescending(a => a.Offset))
                {
                    sb.Insert(piiEntity.Offset + piiEntity.Length, "</b></span>");
                    sb.Insert(piiEntity.Offset, $"<span style='background-color:lightgray;border:1px solid black;corner-radius:2px; color:{GetBackgroundColor(piiEntity)}' title='{piiEntity.Category}: {piiEntity.SubCategory} Confidence: {piiEntity.ConfidenceScore} Redacted Text: {piiEntity.Text}'><b>");
                }
            }
            HtmlRedactedText = sb.ToString()?.CleanupAllWhiteSpace();    
            HtmlRedactedTextMarkupString = new MarkupString(HtmlRedactedText ?? string.Empty);
        }

        private string GetBackgroundColor(PiiEntity piiEntity)
        {
            if (piiEntity.Category == PiiEntityCategory.PhoneNumber)
            {
                return "yellow";
            }
            if (piiEntity.Category == PiiEntityCategory.Organization)
            {
                return "orange";
            }
            if (piiEntity.Category == PiiEntityCategory.Address)
            {
                return "green";
            }
            return "gray";                   
        }

        public long ExecutionTime { get; set; }
        public PiiEntityCollection? AnalysisResult { get; set; }

    }
}




Frontend UI looks like this: Home.razor


@page "/"
@using PiiDetectionDemo.Util

@inject IPiiRemovalTextAnalyticsClientService _piiRemovalTextAnalyticsClientService;

<h3>Azure HealthCare Text Analysis - Pii detection feature - Azure Cognitive Services</h3>

<em>Pii = Person identifiable information</em>

<EditForm Model="@Model" OnValidSubmit="@Submit">
    <DataAnnotationsValidator />
    <ValidationSummary />

    <div class="form-group row">
        <label><strong>Text input</strong></label>
        <InputTextArea @oninput="removeWhitespace" class="overflow-scroll" style="max-height:500px;max-width:900px;font-size: 10pt;font-family:Verdana, Geneva, Tahoma, sans-serif" @bind-Value="@Model.InputText" rows="5" />
    </div>

    <div class="form-group row">
        <div class="col">
            <br />
            <button class="btn btn-outline-primary" type="submit">Run</button>
        </div>
        <div class="col">
        </div>
        <div class="col">
        </div>
    </div>

    <br />

    @if (isProcessing)
    {

        <div class="progress" style="max-width: 90%">
            <div class="progress-bar progress-bar-striped progress-bar-animated"
                 style="width: 100%; background-color: green">
                Retrieving result from Azure Text Analysis Pii detection feature. Processing..
            </div>
        </div>
        <br />

    }

    <div class="form-group row">
        <label><strong>Analysis result</strong></label>

        @if (isSearchPerformed)
        {
            <br />
            <b>Execution time took: @Model.ExecutionTime ms (milliseconds)</b>

            <br />
            <br />

            <b>Redacted text (Pii removed)</b>
            <br />

            <div class="form-group row">
               <label><strong>Categorized Pii redacted text</strong></label>
               <div>
               @Model.HtmlRedactedTextMarkupString
               </div>
            </div>

            <br />
            <br />

            <table class="table table-striped table-dark table-hover">
                <thead>
                <th>Pii text</th>
                <th>Category</th>
                <th>SubCategory</th>
                <th>Offset</th>
                <th>Length</th>
                <th>ConfidenceScore</th>
                </thead>
                <tbody>
                    @if (Model.AnalysisResult != null) {
                        @foreach (var entity in Model.AnalysisResult)
                        {
                            <tr>
                                <td>@entity.Text</td>
                                <td>@entity.Category.ToString()</td>
                                <td>@entity.SubCategory</td>
                                <td>@entity.Offset</td>
                                <td>@entity.Length</td>
                                <td>@entity.ConfidenceScore</td>                                        
                            </tr>
                        }
                    }
                </tbody>
            </table>

        }
    </div>

</EditForm>



The Demo uses Bootstrap 5 to build up a HTML table styled and showing the Azure.AI.TextAnalytics.PiiEntity properties.

Sunday 14 April 2024

Building a filter via Expression trees in C#

This article will look at how to build a filter with Expression trees in C#.

It is an academic exercise how to use Expression trees, you would probably use filters just specifying lambda function with LINQ, but the code shows how you can build an Expression incrementally and compile it to a function. If there is a use-case where LINQ does not fit, perhaps some late-binding scenario or where LINQ does not offer an operator, you can use the approach shows in this article, but the article shows simple usage of Expression trees for introducing Expression trees to C# developers wanting to
test them out in more detail.

The sample code below shows the sample code testing out how to build the Expression incrementally using extension method loading some sample data. An important gotcha is to keep sending in same the objectParameter which is the parameter expression used in the lambda function that is built up, this must be the same parameter. Consider some lambda function of an object 'Person' where the parameter 'x' like:

x => x.Age > 3 && x.Age < 9

The point is that the ParameterExpression x must be the same object, or else we get an error.

FilterHelper.cs


public static class FilterHelper {

	public enum ComparisonOperator {
		Equal,
		LessThan,
		LessThanOrEqual,
		GreaterThan,
		GreaterThanOrEqual,
		NotEqual		
	}
	
	public static Func<TClass, bool> CompileFilter<TClass>(this Expression expression, ParameterExpression objectParameter){
		var expr = Expression.Lambda<Func<TClass, bool>>(expression, false, new List<ParameterExpression>{ objectParameter });
		return expr.Compile();
	}
	
	public static Expression BuildFilter<TClass, TProp>(this Expression previousExpression, Expression<Func<TClass, TProp>> prop, object value,
		ComparisonOperator op, ParameterExpression objectParameter){
		
		var propertyInfo = GetPropertyInfo(prop);
		var propertyToCall = Expression.Property(objectParameter, propertyInfo);		
		var valueToTest = Expression.Constant(value);
		
		Expression operatorExpression = null;
		switch (op)
		{
			case ComparisonOperator.Equal:
				operatorExpression = Expression.Equal(propertyToCall, valueToTest);
				break;
			case ComparisonOperator.NotEqual:
				operatorExpression = Expression.NotEqual(propertyToCall, valueToTest);
				break;
			case ComparisonOperator.LessThan:
				operatorExpression = Expression.LessThan(propertyToCall, valueToTest);
				break;
			case ComparisonOperator.LessThanOrEqual:
				operatorExpression = Expression.LessThanOrEqual(propertyToCall, valueToTest);
				break;
			case ComparisonOperator.GreaterThan:
				operatorExpression = Expression.GreaterThan(propertyToCall, valueToTest);
				break;
			case ComparisonOperator.GreaterThanOrEqual:
				operatorExpression = Expression.GreaterThanOrEqual(propertyToCall, valueToTest);
				break;
		}
		
		if (previousExpression == null){
			return operatorExpression;
		}
		else {
			return Expression.AndAlso(previousExpression, operatorExpression);
		}			
	}

	/// <summary>
	/// Gets the corresponding <see cref="PropertyInfo" /> from an <see cref="Expression" />.
	/// </summary>
	/// <param name="property">The expression that selects the property to get info on.</param>
	/// <returns>The property info collected from the expression.</returns>
	/// <exception cref="ArgumentNullException">When <paramref name="property" /> is <c>null</c>.</exception>
	/// <exception cref="ArgumentException">The expression doesn't indicate a valid property."</exception>
	private static PropertyInfo GetPropertyInfo<T, P>(Expression<Func<T, P>> property)
	{
		if (property == null)
		{
			throw new ArgumentNullException(nameof(property));
		}

		if (property.Body is UnaryExpression unaryExp)
		{
			if (unaryExp.Operand is MemberExpression memberExp)
			{
				return (PropertyInfo)memberExp.Member;
			}
		}
		else if (property.Body is MemberExpression memberExp)
		{
			return (PropertyInfo)memberExp.Member;
		}

		throw new ArgumentException($"The expression doesn't indicate a valid property. [ {property} ]");
	}

}




The sample data uses a POCO Employee as entity class: Employee.cs


public class Employee
{
	public int Id { get; set; }
	public string FirstName { get; set; }
	public string LastName { get; set; }
	public string Department { get; set; }
	public string Position { get; set; }
	public decimal Salary { get; set; }
	public DateTime HireDate { get; set; }
	public DateOnly HireDateOnly
	{
		get { return DateOnly.FromDateTime(HireDate); }
	}
}





The sample program loads up the Json data, then it builds the expression with method BuildFilter shown above and then finally calls CompileFilter to build the expression into a Func<TClass, bool> where TClass is the employee type.

Program.cs


void Main()
{
	string json = File.ReadAllText(Path.Combine(@"C:\Users\SomeUser\Documents\LINQPad Queries\SampleData\Employees.json"));
	var employees = JsonSerializer.Deserialize<List<Employee>>(json, 
new JsonSerializerOptions { PropertyNameCaseInsensitive = true }); //employees.Dump(); var objectParameter = Expression.Parameter(typeof(Employee)); Expression currentFilter = null; currentFilter = FilterHelper.BuildFilter<Employee, object>(currentFilter, e => e.Department,
"Engineering", FilterHelper.ComparisonOperator.Equal, objectParameter) .BuildFilter<Employee, object>(e => e.Salary, 79000m,
FilterHelper.ComparisonOperator.GreaterThan, objectParameter); Func<Employee, bool> employeeFilter = currentFilter.CompileFilter<Employee>(objectParameter); var matchingEmployees = employees.Where(employeeFilter).ToList(); matchingEmployees.Dump(); }


Sample data json looks like this - an array of employees

Employees.json


[
    {
      "id": 1,
      "firstName": "Alice",
      "lastName": "Johnson",
      "department": "HR",
      "position": "Manager",
      "salary": 60000,
      "hireDate": "2022-03-15"
    },
    {
      "id": 2,
      "firstName": "Bob",
      "lastName": "Smith",
      "department": "Engineering",
      "position": "Software Engineer",
      "salary": 80000,
      "hireDate": "2021-09-10"
    },
    {
      "id": 3,
      "firstName": "Charlie",
      "lastName": "Brown",
      "department": "Finance",
      "position": "Financial Analyst",
      "salary": 70000,
      "hireDate": "2020-05-20"
    },
    {
      "id": 4,
      "firstName": "David",
      "lastName": "Lee",
      "department": "Marketing",
      "position": "Marketing Specialist",
      "salary": 65000,
      "hireDate": "2019-11-05"
    },
    {
      "id": 5,
      "firstName": "Eva",
      "lastName": "Garcia",
      "department": "Sales",
      "position": "Sales Representative",
      "salary": 75000,
      "hireDate": "2018-07-12"
    },
    {
      "id": 6,
      "firstName": "Frank",
      "lastName": "Wang",
      "department": "Engineering",
      "position": "Senior Developer",
      "salary": 95000,
      "hireDate": "2017-02-28"
    },
    {
      "id": 7,
      "firstName": "Grace",
      "lastName": "Miller",
      "department": "HR",
      "position": "Recruiter",
      "salary": 55000,
      "hireDate": "2016-08-18"
    },
    {
      "id": 8,
      "firstName": "Henry",
      "lastName": "Chen",
      "department": "Finance",
      "position": "Financial Manager",
      "salary": 90000,
      "hireDate": "2015-04-03"
    },
    {
      "id": 9,
      "firstName": "Ivy",
      "lastName": "Nguyen",
      "department": "Marketing",
      "position": "Content Writer",
      "salary": 60000,
      "hireDate": "2014-10-22"
    },
    {
      "id": 10,
      "firstName": "Jack",
      "lastName": "Kim",
      "department": "Sales",
      "position": "Account Executive",
      "salary": 80000,
      "hireDate": "2013-06-14"
    },
    {
      "id": 11,
      "firstName": "Karen",
      "lastName": "Taylor",
      "department": "Engineering",
      "position": "QA Engineer",
      "salary": 75000,
      "hireDate": "2012-01-09"
    },
    {
      "id": 12,
      "firstName": "Leo",
      "lastName": "Rodriguez",
      "department": "HR",
      "position": "HR Specialist",
      "salary": 55000,
      "hireDate": "2011-07-27"
    },
    {
      "id": 13,
      "firstName": "Mia",
      "lastName": "Liu",
      "department": "Finance",
      "position": "Financial Advisor",
      "salary": 70000,
      "hireDate": "2010-03-16"
    },
    {
      "id": 14,
      "firstName": "Nina",
      "lastName": "Martinez",
      "department": "Marketing",
      "position": "Social Media Manager",
      "salary": 65000,
      "hireDate": "2009-09-05"
    },
    {
      "id": 15,
      "firstName": "Oscar",
      "lastName": "Hernandez",
      "department": "Sales",
      "position": "Sales Manager",
      "salary": 100000,
      "hireDate": "2008-04-21"
    }
  ]



The filter is more limited than just sticking to LINQ, but the code in this example shows how you can build a filter incrementally. Traditionally, you would use Linq and an IEnumerable of TClass and you can keep on filter it too. Here are some closing arguments for why you could make use of Expression trees and have to use them too and not be able to use Linq:
Purpose: Expression trees represent code as data structures. They allow you to build executable code dynamically in C#. Use Cases: - Dynamic Code Generation: When you need to create or modify code at runtime (e.g., building custom queries or transformations). - Remote Execution: Expression trees are useful for scenarios where you want to send calculations across the wire (e.g., database queries, web services). - Custom Query Providers: If you’re building your own query provider (like LINQ to SQL or Entity Framework), expression trees help translate queries into other formats (e.g., SQL).

Wednesday 27 March 2024

Importing Json File to SQL Server into a variable

A short article today of how to import JSON file to SQL Server into a variable, which can then
be used to insert it into a column of type NVARCHAR(MAX) of a table. The maximum size of NVARCHAR(MAX) is 2 Gb, so you can
import large Json files using this datatype. If the Json is small and below 4000 chars, use for example NVARCHAR(4000) instead. Here is a SQL script to import the json file using OPENROWSET and Bulk import. We also pass in the path to the folder where the json file is. It is put in the same folder as the .sql file script. Note that the variable $(FullScriptDir) is passed in via a .bat file (shown further below) and we expect the .json file to be in the same folder as the .bat file. You can provide a full path to a .json file instead and skip the .bat file here and import a json file, but it is nice to load the .json file from the same folder as the .sql file in case you want to copy the .sql and .json file to another server and not having to provide and possibly having to adjust the full path. Sql-script import_json_file_openrowset.sql:


DECLARE @JSONFILE VARCHAR(MAX); 

SELECT @JSONFILE = BulkColumn
FROM OPENROWSET (BULK '$(FullScriptDir)\top-posts.json', SINGLE_CLOB) AS j;

PRINT 'JsonFile contents: ' + @JSONFILE

IF (ISJSON(@JSONFILE)=1) PRINT 'It is valid Json';


The .bat file here passes the current folder as a variable to the sql script runsqlscript.bat


@set FullScriptDir=%CD%
sqlcmd -S .\SQLEXPRESS  -i import_json_file_openrowset.sql


This outputs:


sqlcmd -S .\SQLEXPRESS  -i import_json_file_openrowset.sql
JsonFile contents: [
   {
      "Id":6107,
      "Score":176,
      "ViewCount":155988,
      "Title":"What are deconvolutional layers?",
      "OwnerUserId":8820
   },
   {
      "Id":155,
      "Score":164,
      "ViewCount":25822,
      "Title":"Publicly Available Datasets",
      "OwnerUserId":227
   }
]
It is valid Json


With the variable JSONFILE you can do whatever with it such as inserting it to a column in a new row of a table for example.
Importing json from a string directly using OPENJSON

It is also possible to directly just import the JSON from a string variable like this:


DECLARE @JSONSTRINGSAMPLE VARCHAR(MAX) 

SET @JSONSTRINGSAMPLE = N'[
 {
    "Id": 2334,
    "Score": 4.3,
    "Title": "Python - Used as scientific tool for graphing"
 },
{
    "Id": 2335,
    "Score": 5.2,
    "Title": "C# : Math and physics programming"
 }
]';

SELECT * FROM OPENJSON (@JSONSTRINGSAMPLE) WITH (
    Id INT,
    Score REAL,
    Title NVARCHAR(100)
)


Tuesday 19 March 2024

Functional programming - Fork combinator in C# to combine results from parts

This article will discuss a wellknown combinator called Fork which allows you to combine the mapped result. Consider the following extension methods to fork on an object. Fork here means to operate on parts of the object such as
different properties and apply functions on these parts and then recombine the results into a combined result via a specified combinator function, sometimes called a 'join function'.


public static class FunctionalExtensions {

	public static TOutput Map<TInput, TOutput>(
		this TInput @this,
		Func<TInput, TOutput> func) => func(@this);

	public static TOutput Fork<TInput, TMiddle, TOutput>(
		this TInput @this,
		Func<IEnumerable<TMiddle>, TOutput> combineFunc,
		params Func<TInput, TMiddle>[] parts)
	{
		var intermediateResults = parts.Select(p => p(@this));
		var result = combineFunc(intermediateResults);
		return result;
    }

	public static TOutput Fork<TInput, TMiddle, TOutput>(
		this TInput @this,
		Func<TInput, TMiddle> leftFunc,
		Func<TInput, TMiddle> rightFunc,
		Func<TMiddle, TMiddle, TOutput> combineFunc)
	{
		var leftResult = leftFunc(@this); // @this.Map(leftFunc);
		var rightResult = rightFunc(@this); // @this.Map(rightFunc);
		var combineResult = combineFunc(leftResult, rightResult);
		return combineResult;
	}

}


Let's take a familiar mathematical example, calculating the Hypotenuse in a triangle using Pythagorean theorem. This states that the length of the longest side A of a 'right triangle' is the square root of the sum of the squares of the shorter sides B and C : A = √(B² + C²) Consider this class:
  
  
  public class Triangle {
	public double CathetusA { get; set; }
	public double CathetusB { get; set; }	
	public double Hypotenuse { get; set; }
  }
  
    
Let's test the first Fork helper extension method accepting two functions for specifying the left and right components:
  
  
  	var triangle = new Triangle
	{
		CathetusA = 3,
		CathetusB = 4
	};
	
	triangle.Hypotenuse = triangle.Fork(	
		t => t.CathetusA * t.CathetusA, 
		t => t.CathetusB * t.CathetusB, 
		(l,r) => Math.Sqrt(l+r));
		
	Console.WriteLine(triangle.Hypotenuse);
  
  
  
This yields '5' as the answer via the forked result above. A simple example, but this allows us to create a simple combinatory logic example on an object of any type using functional programming (FP). Let's look at a simpler example just combining multiple properties of an object with a simple string-join, but using the Fork version supporting arbitrary number of parts / components:
 


public class Person {
	public string JobTitle { get; set; }
	public string FirstName { get; set; }
	public IEnumerable<string> MiddleNames { get; set; }
	public string LastName { get; set; }
}

var person = new Person{
		JobTitle = "Detective",
		FirstName = "Alexander",
		MiddleNames = new[] { "James", "Axel" },
		LastName = "Foley"
	};
	
string contactCardText = person.Fork(parts => string.Join(" ", parts), p => p.FirstName,
p => string.Join(" ", p.MiddleNames), p => p.LastName); Console.WriteLine(contactCardText);
This yields: Alexander James Axel Foley Fork can be very useful in many cases you need to 'branch off' on an object and recombine parts of the object with some specific function, either two parts or multiple parts and either continue to work on the results or retrieve the results.

Sunday 10 March 2024

Functional programming - the Tee function to inspect current state in a chained expression

In this article we will look at helper extension methods of StringBuilder first to better support chaining StringBuilder. We will work on the same StringBuilder instance and add support for appending lines or character to the StringBuilder given a condition. Also example showing how to aggregate lines from a sequence is shown and appending formatted lines. Since C# interpolation has become more easy to use, I would suggest you keep using AppendLine instead. Here is the helper methods in the extension class :


public static class StringBuilderExtensions {

	public static StringBuilder AppendSequence<T>(this StringBuilder @this, IEnumerable<T> sequence, Func<StringBuilder, T, StringBuilder> fn)
	{
		var sb = sequence.Aggregate(@this, fn);
		return sb;
	}
	
	public static StringBuilder AppendWhen(this StringBuilder @this, Func<bool> condition, Func<StringBuilder, StringBuilder> fn) => 
		condition() ? fn(@this) : @this;
		
    public static StringBuilder AppendFormattedLine(
		this StringBuilder @this,
		string format,
		params object[] args) => 
			@this.AppendFormat(format, args).AppendLine();
	
}


Now consider this example usage:


void Main()
{
	var countries = new Dictionary<int, string>{
		{ 1, "Norway" },
		{ 2, "France" },
		{ 3, "Austria" },
		{ 4, "Sweden" },
		{ 5, "Finland" },
		{ 6, "Netherlands" }
	};
	string options = BuildSelectBox(countries, "countriesSelect", true);
	options.Dump("Countries"); //dump is a method available in Linqpad to output objects 
	
}

private static string BuildSelectBox(IDictionary<int, string> options, string id, bool includeUnknown) =>
		new StringBuilder()
			.AppendFormattedLine($"<select id=\"{id}\" name=\"{id}\">")
			.AppendWhen(() => includeUnknown, sb => sb.AppendLine("\t<option value=\"0\">Unknown</option>"))
			.AppendSequence(options, (sb, item) => sb.AppendFormattedLine("\t<option value=\"{0}\">{1}</option>", item.Key, item.Value))
			.AppendLine($"</select>").ToString();   


What if we wanted to inspect the state of the stringbuilder in the middle of these chained expression. Is it possible to output state in such lengthy chained functional expressions? Yes, that is called the Tee method inside functional programming patterns. Other might call it for Tap such as used in Rx languages. The Tee method looks like this:
 
 
public static class FunctionalExtensions {

	public static T Tee<T>(this T @this, Action<T> act) {
		act(@this);
		return @this;
	}
	
}

 
We can now inspect state in the middle of chained expressions in functional expressions.
 
 
 
private static string BuildSelectBox(IDictionary<int, string> options, string id, bool includeUnknown) =>
		new StringBuilder()
			.AppendFormattedLine($"<select id=\"{id}\" name=\"{id}\">")
			.AppendWhen(() => includeUnknown, sb => sb.AppendLine("\t<option value=\"0\">Unknown</option>"))
            .Tee(Console.WriteLine)
			.AppendSequence(options, (sb, item) => sb.AppendFormattedLine("\t<option value=\"{0}\">{1}</option>", item.Key, item.Value))
			.AppendLine($"</select>").ToString();   
 
 
The picture below shows the output:
So there you have it, if you have lengthy chained functional expressions, make such a Tee helper method to peek into the state this far. The name Tee stems from the Unix Command by the same name. It copies contents from STDIN to STDOUT. More about Tee Unix command here:
https://shapeshed.com/unix-tee/

Saturday 9 March 2024

Functional programming - looking up current time and encapsulating usings

I looked at encapsulating Using statements today for functional programming and how to look up the current time with API available on the Internet.


public static class Disposable {
	
	public static TResult Using<TDisposable,TResult>(
		Func<TDisposable> factory,
		Func<TDisposable, TResult> map)		
		where TDisposable : IDisposable
	{
		using (var disposable = factory()){
			return map(disposable);
		}
		
	}	
}

void Main()
{
	var currentTime = EpochTime.AddSeconds(Disposable
			  .Using(() => new HttpClient(),
					client => JsonDocument.Parse(client.GetStringAsync(@"http://worldtimeapi.org/api/timezone/europe/oslo").Result))
			  .RootElement
			  .GetProperty("unixtime")
			 .GetInt64()).ToLocalTime(); //list of time zones available here: http://worldtimeapi.org/api/timezone
	currentTime.Dump("CurrentTime");	
}

public static DateTime EpochTime => new DateTime(1970, 1, 1);



The Disposable is abstracted away in the helper method called Using accepting a factory function to create a TDisposable that accepts an IDisposable. We look up the current time using the WorldTimeApi and make use of extracting the UnixTime which is measured from Epoch as the number of seconds elapsed from 1st January 1970. We make use of System.Text.Json here, which is part of .NET to parse the json retrieved.

Thursday 7 March 2024

Currying functions in C#

This article will look into helper methods for currying functions in C#. The definition of Currying consists of splitting up a function with multiple arguments into multiple functions accepting one argument. But you can also have some of the arguments provided via smaller functions, so be aware also of this alternative. What is in the name currying? The name has nothing to do with cooking from India, but comes from the mathematician Haskell Brooks Curry (!)

https://en.wikipedia.org/wiki/Haskell_Curry

A reason for introducing support for currying is that you can build complex functions from simpler functions as building blocks. Currying is explained great here:
https://www.c-sharpcorner.com/UploadFile/rmcochran/functional-programming-in-C-Sharp-currying/

We will see in the examples that we can provide multiple arguments at once and the syntax will look a bit special compared to other C# code. Curryings benefits is to allow a more flexible way to call a method. You can store into variables calls to a function providing a subset of argument and use that variable to either specify an intermediate other call or get the final result. Note - The function will be called when ALL arguments are provided ONCE ! This helps a lot of avoiding surprising side effects. Let's first look at a sample set of methods we want to support currying.


int FooFourArgs(string st, float x, int j, int k)
{
	Console.WriteLine($"Inside method FooFourArgs. Got parameters: st={st}, x={x}, j={j}, k={k}");
	return 42;
}

int FooThreeArgs(string st, float x, int j)
{
	Console.WriteLine($"Inside method FooThreeArgs. Got parameters: st={st}, x={x}, j={j}");
	return 42;
}

int FooTwoArgs(string st, float x)
{
	Console.WriteLine($"Inside method FooTwoArgs. Got parameters: st={st}, x={x}");
	return 41;
}

int FooOneArgs(string st)
{
	Console.WriteLine($"Inside method FooOneArgs. Got parameters: st={st}");
	return 40;
}


We want to call the sample methods above in a more flexible way by splitting the number of arguments we provide. Let's see the extension methods to call up to four arguments to a function. Note the use of chaining the lambda operator (=>) to provide the support for currying.


public static class FunctionExtensions
{
	public static Func<T1, TResult> Curried<T1, TResult>(this Func<T1, TResult> func)
	{
		return x1 => func(x1);
	}
	
	public static Func<T1, Func<T2, TResult>> Curried<T1, T2, TResult>(this Func<T1, T2, TResult> func)
	{
		return x1 => x2 => func(x1, x2);
	}

	public static Func<T1, Func<T2, Func<T3, TResult>>> Curried<T1, T2, T3, TResult>(this Func<T1, T2, T3, TResult> func)
	{
		return x1 => x2 => x3 => func(x1, x2, x3);
	}

	public static Func<T1, Func<T2, Func<T3, Func<T4, TResult>>>> Curried<T1, T2, T3, T4, TResult>(this Func<T1, T2, T3, T4, TResult> func)
	{
		return x1 => x2 => x3 => x4 => func(x1, x2, x3,x4);
	}
}


The following main method shows how to use these curry helper methods:


void Main()
{
	var curryOneArgsDelegate = new Func<string, int>((st) => FooOneArgs(st)).Curried();
	var curryOneArgsPhaseOne = curryOneArgsDelegate("hello");

	var curryTwoArgsDelegate = new Func<string, float, int>((st, x) => FooTwoArgs(st,x)).Curried();
	var curryTwoArgsPhaseOne = curryTwoArgsDelegate("hello");
	var curryTwoArgsPhaseTwo = curryTwoArgsPhaseOne(3.14f);

	var curryThreeArgsDelegate = new Func<string, float, int, int>((st, x, j) => FooThreeArgs(st, x, j)).Curried();
	var curryThreeArgsPhaseOne = curryThreeArgsDelegate("hello");
	var curryThreeArgsPhaseTwo = curryThreeArgsPhaseOne(3.14f);
	var curryThreeArgsPhaseThree = curryThreeArgsPhaseTwo(123);	
	//Or call currying in a single call passing in two or more parametres
	var curryThreeArgsPhaseOneToThree = curryThreeArgsDelegate("hello")(3.14f)(123);

	var curryFourArgsDelegate = new Func<string, float, int, int, int>((st, x, j, k) => FooFourArgs(st, x, j, k)).Curried();
	var curryFourArgsPhaseOne = curryFourArgsDelegate("hello");
	var curryFourArgsNextPhases = curryFourArgsPhaseOne(3.14f)(123)(456); //just pass in the last arguments if they are known at this stage
	curryFourArgsDelegate("hello")(3.14f)(123)(456); //you can pass in 1-4 parameters to FooFourArgs method - all in a single call for example or one by one
}


The output we get is this. Note that we only call the methods we defined when all parameters are sent in. The function call which had partial argument list provided did not result into a function call.


Inside method FooOneArgs. Got parameters: st=hello
Inside method FooTwoArgs. Got parameters: st=hello, x=3,14
Inside method FooThreeArgs. Got parameters: st=hello, x=3,14, j=123
Inside method FooThreeArgs. Got parameters: st=hello, x=3,14, j=123
Inside method FooFourArgs. Got parameters: st=hello, x=3,14, j=123, k=456


So from a higher level, currying a function f(x,y,z) means adding support that you could call the function like this:
f(x,g(y,z)) or f(x,g(y,h(z))) - there more arguments you get there is more variations of number of parameters and methods you can pass in. Here is another example how you can build up a calculation uing simpler methods.


void Main()
{
	Func Area = (x,y) => x*y;
	Func CubicArea = (x,y,z) => Area.Curried()(Area(x,y))(z);	
	CubicArea(3,2,4); //supplying all arguments manully is okay
}


CubicArea expects THREE arguments. The implementation allows us to use the Area function and via currying we can use that method and provide the last third argument avoiding compilation error. Currying makes your functions allow more flexible ways of being called.

Saturday 24 February 2024

Using IronPython to execute Python code from .NET

Let's look at some code showing how to execute Python code from .NET using IronPython! IronPython provides support for Python scripts to run inside .NET and utilizes the Dynamic Language Runtime - DLR. The DLR together allows the caller to get dynamic typing and dynamic method dispatch, which is central in the dynamic languages such as Python. IronPython was first released in 2004, some 20 years ago. It has continued to evolve slowly and provides seamless integration into .NET ecosystem for Python developers. In this article, I will present some simple code that shows how you can run Python code inside a .NET 8 console application. We will load up some tuples in an array in some simple Python code, using IronPython. Tuples in Python Tuples in Python are immutable (such as in C#) and are defined using parentheses and comma-separated. This is the same as in C#, but Python had tuple support over 20 years before C#. We will have to add one Nuget package, the IronPython package, in a net8.0 application.

HelloIronPythonDemo1.csproj



<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="IronPython" Version="3.4.1" />
  </ItemGroup>

  <ItemGroup>
    <None Update="customers.py">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
  </ItemGroup>

</Project>



Consider the following array of tuples in Python :

customers.py

customers = [
	('Jenna', 42, 165),
	('Thor', 40, 174),
	('Christopher', 18, 170),
	('Liz', 16, 168),
 ]

Python code is very compact and you declare variables without specifying type such as in C#, Python uses a simple way of creating variables and while C# got support in C# 7 in 2017, Python has had support for tuples since its early days. In the Python 1.4 version, we find it documented here:
https://docs.python.org/release/1.4/tut/node37.html#SECTION00630000000000000000.
Bear in mind, this is way back in 1996, C# was over 20 years later with its tuple support. If you install IronPython, you get a terminal where you can enter Python code (plus more functionality with .NET) such as shown below, where tuples are created and tuples may be composed or 'packed' and also 'unpacked', which is called deconstructed in .NET tuples.
To execute code to retrieve this array of tuples, first create a ScriptEngine and then create a ScriptScope, which we will use to retrieve the Python-declared variable customers. We create a ScriptSource, where we use the ScriptEngine to load up either a string or a file. A dynamic variable will be used to get the array of tuples and we can loop through this array with a foreach loop and output its content.

Program.cs
 
 using IronPython.Hosting;
 using Microsoft.Scripting.Hosting;
 using static System.Console;   
 IronPythonDemo1.OutputSomeExternallyLoadedTuples();

 public class IronPythonDemo1
 {

    public static void OutputSomeExternallyLoadedTuples()
    {
        var engine = Python.CreateEngine();
        ScriptScope scope = engine.CreateScope();

        //ScriptSource source = engine.CreateScriptSourceFromString(tupleStatement);
        ScriptSource source = engine.CreateScriptSourceFromFile("customers.py");
        source.Execute(scope);
        dynamic customers = scope.GetVariable("customers");
        foreach (var customer in customers)
        {
            Console.WriteLine($"(Name = {StringExtensions.FixedLength(customer[0], 20)}, Age = {StringExtensions.FixedLength(customer[1].ToString(), 8)}, Height={StringExtensions.FixedLength(customer[2].ToString(), 8)})");
        }
    }

}
 
 
 
Documentation for named tuples are available here: https://docs.python.org/3/library/collections.html#collections.namedtuple Here is sample coding showing script that although it is more verbose, shows more readability of which field is which for a named tuple. In an ordinary tuple, you use indexes to retrieve the nth field (0-based). But with named tuples, you use a field name instead.
 
from collections import namedtuple

Customer = namedtuple('Customer', ['Name', 'Age', 'Height'])
 
customers2 =  [
    Customer(Name = 'Jenna', Age = 42, Height = 165),
    Customer(Name = 'Thor', Age = 38, Height = 174),
    Customer(Name = 'Christopher', Age = 42, Height = 170),
    Customer(Name = 'Liz', Age = 42, Height = 168),
 ]
   
for cust in customers2:
    print(f"{cust.Name} with a height of {cust.Height}(cm)")

This outputs:
 
Jenna with a height of 165(cm)
Thor with a height of 174(cm)
Christopher with a height of 170(cm)
Liz with a height of 168(cm)
When your tuple gets many fields, having this readability should reduce bugs. Also, if you add more fields to your tuple, you do not have to fix up indexes in your script. So code is a bit more verbose, but it is also more open for change and readable.

The FixedLength extension method is a simple method to output text to a fixed width.
 
 
 public static class StringExtensions
 {

    public static string FixedLength(this string input, int length, char paddingchar = ' ')
    {
        if (string.IsNullOrWhiteSpace(input))
        {
            return input;
        }
        if (input.Length > length)
            return input.Substring(0, length);
        else
            return input.PadRight(length, paddingchar);
    }

 } 
 

Thursday 1 February 2024

Creating a data table from IEnumerable of T and defining column order explicitly in C#

This article shows code how you can create a DataTable from a collection of T (IEnumerable<T>) and defining explicitly the column order. An extension method for this looks like the following:



public static class DataTableExtensions
{


	public static DataTable CreateOrderedDataTable<T>(this IEnumerable<T> data)
	{
		var dataTable = new DataTable();
		var orderedProps = typeof(T).GetProperties(BindingFlags.Instance | BindingFlags.Public)
		 .OrderBy(prop => GetColumnOrder(prop)).ToList();
		
		foreach (var prop in orderedProps){
			dataTable.Columns.Add(prop.Name, Nullable.GetUnderlyingType(prop.PropertyType) ?? prop.PropertyType);
		}
		
		if (data != null)
		{
			dataTable.BeginLoadData();
			var enumerator = data.GetEnumerator();
			while (enumerator.MoveNext()){
			   var item = enumerator.Current;
			   var rowValues = new List<object>();
			   foreach (var prop in orderedProps){
			    rowValues.Add(prop.GetValue(item, null));		   	
			   }
			   dataTable.Rows.Add(rowValues.ToArray());			 			
			}
			dataTable.AcceptChanges();
		}
		return dataTable;
	}

	static int GetColumnOrder(PropertyInfo prop)
	{
		var displayAttribute = prop.GetCustomAttributes(typeof(DisplayAttribute), false).FirstOrDefault() as DisplayAttribute;
		int orderKey = displayAttribute?.Order ?? prop.MetadataToken;		
		return orderKey;
	}
	
}



We order first by DisplayAttribute and the Order value, and fallback to property's MetadataToken. This is an integer value that also returns the order the property was declared, in case you want to order just by the way properties are defined. We get the enumerator here and fetch the row one by one. We could use a simple foreach loop here too. Note the use of BeginLoadData and AcceptChanges. Consider the two classes next. One class does not set any explicit order, the other class uses the Display attribute's Order value to define a custom order of columns for the DataTable.


public class Car
{

	public int Id { get; set; }

	public string Make { get; set; }

	public string Model { get; set; }

	public string Color { get; set; }
}


public class CarV2
{
	[Display(Order = 4)]
	public int Id { get; set; }
	
	[Display(Order = 3)]
	public string Make { get; set; }
	
	[Display(Order = 2)]
	public string Model { get; set; }

	[Display(Order = 14)]
	public bool IsElectric { get; set; }

	[Display(Order = -188865)]
	public string Color { get; set; }
	
}


Next, the following little program in Linqpad tests this extension method and displays the datatables resulting with column ordering set.



void Main()
{
	var cars = new List<Car>{
		new Car { Id = 1, Make = "Audi", Model = "A5", Color = "Blue" },
		new Car { Id = 2, Make = "Volvo", Model = "XC60", Color = "Silver" },
		new Car { Id = 3, Make = "Golf", Model = "GTI", Color = "White" },
		new Car { Id = 4, Make = "Audi", Model = "A5", Color = "Blue" },
	};
	var dataTable = cars.CreateOrderedDataTable();
	dataTable.Dump("Cars datatable, data type is: Car");
	
	var carV2s = new List<CarV2>{
		new CarV2 { Id = 1, Make = "Audi", Model = "A5", Color = "Blue" },
		new CarV2 { Id = 2, Make = "Volvo", Model = "XC60", Color = "Silver" },
		new CarV2 { Id = 3, Make = "Golf", Model = "GTI", Color = "White" },
		new CarV2 { Id = 4, Make = "Audi", Model = "A5", Color = "Blue" },
	};	
	var dataTableV2 = carV2s.CreateOrderedDataTable();
	dataTableV2.Dump("Carsv2 datatable, datatype is CarV2");

}


Sunday 14 January 2024

Generating repeated data into variable in SQL Server in T-SQL

Let's see how we can create repeated data into variable of SQL Server in T-SQL. Use the REPLICATE function to create repeated data like this:


DECLARE @myVariable NVARCHAR(MAX)
SET @myVariable = REPLICATE('.', 10)
PRINT @myVariable
PRINT len(@myVariable)





In case you want to set the variable to data which is longer than 8000 characters, you must convert the argument to NVARCHAR(MAX).


DECLARE @myVariable NVARCHAR(MAX)
SET @myVariable = REPLICATE(CONVERT(NVARCHAR(MAX),'.'), 1024*1024*2)
PRINT len(@myVariable)


Creating random content is also easy in T-SQL:

DECLARE @myVariable NVARCHAR(MAX)
SET @myVariable = REPLICATE(CONVERT(NVARCHAR(MAX),REPLACE(NEWID(),'-', '')), 4)
PRINT len(@myVariable)
PRINT @myVariable

NEWID() creates a new guid, and we strip away the '-' letter, giving 32 chars which we replicate above four times. Since we were below 8000 chars, we chould have skipped using convert to nvarchar(max).