Monday, 15 July 2024

Caching pure functions using Memoize in C#

This article will present a technique for caching pure functions in C# using Memoize technique. This is a programmatic caching of pure method or function where we have a method that always returns the same result or equivalent result given an input. This adds scalability, maybe the method takes long time to process and we want to avoid using resources and provide a quicker answer. If your method has side effects or does not yield the same or equivalent result (cosmetic changes ignored) given a set of parameter(s), it should not be memoized. But if it does, here is how you can do this. Note that memoize is a general technique used in functional programming and is used in many languages such as Javascript, for example in the Underscore.Js lib. First off, let's define some POCOs to test the memoize function out. We will use a small sample set of movies and their actors and additional information from the fabulous year 1997.

MovieStore.cs


public class MovieStore {
    public string GetActorsByMovieTitle(string movieTitle)
    {
        Console.WriteLine($"Retrieving actors for movie with title {movieTitle} at: {DateTime.Now}");
        List<Movie> movies1997 = System.Text.Json.JsonSerializer.Deserialize<List<Movie>>(movies1997json);
        string actors = string.Join(",", movies1997
        	.FirstOrDefault(m => m.name?.ToLower() == movieTitle?.ToLower())?.actors.ToArray());
        return actors;
    }   
    
    string movies1997json = """
[
{
  "name": "The Lost World: Jurassic Park",
  "year": 1997,
  "runtime": 129,
  "categories": [
    "adventure",
    "action",
    "sci-fi"
  ],
  "releasedate": "1997-05-23",
  "director": "Steven Spielberg",
  "writer": [
    "Michael Crichton",
    "David Koepp"
  ],
  "actors": [
    "Jeff Goldblum",
    "Julianne Moore",
    "Pete Postlethwaite"
  ],
  "storyline": "Four years after the failure of Jurassic Park on Isla Nublar, John Hammond reveals to Ian Malcolm that there was another island (\"Site B\") on which dinosaurs were bred before being transported to Isla Nublar. Left alone since the disaster, the dinosaurs have flourished, and Hammond is anxious that the world see them in their \"natural\" environment before they are exploited."
},
{
  "name": "The Fifth Element",
  "year": 1997,
  "runtime": 127,
  "categories": [
    "action",
    "adventure",
    "sci-fi"
  ],
  "releasedate": "1997-05-09",
  "director": "Luc Besson",
  "writer": [
    "Luc Besson",
    "Robert Mark Kamen"
  ],
  "actors": [
    "Bruce Willis",
    "Milla Jovovich",
    "Gary Oldman",
    "Chris Tucker",
    "Ian Holm",
    "Luke Perry",
    "Brion James",
    "Tommy Lister",
    "Lee Evans",
    "Charlie Creed-Miles",
    "John Neville",
    "John Bluthal",
    "Mathieu Kassovitz",
    "Christpher Fairbank"
  ],
  "storyline": "In the colorful future, a cab driver unwittingly becomes the central figure in the search for a legendary cosmic weapon to keep Evil and Mr. Zorg at bay."
} ,
{
  "name": "Starship Troopers",
  "year": 1997,
  "runtime": 129,
  "categories": [
    "action",
    "adventure",
    "sci-fi",
    "thriller"
  ],
  "releasedate": "1997-11-07",
  "director": "Paul Verhoeven",
  "writer": [
    "Edward Neumeier",
    "Robert A. Heinlein"
  ],
  "actors": [
    "Casper Van Dien",
    "Dina Meyer",
    "Denise Richards",
    "Jake Busey",
    "Neil Patrick Harris",
    "Clancy Brown",
    "Seth Gilliam",
    "Patrick Muldoon",
    "Michael Ironside"
  ],
  "storyline": "In the distant future, the Earth is at war with a race of giant alien insects. Little is known about the Bugs except that they are intent on the eradication of all human life. But there was a time before the war... A Mobile Infantry travels to distant alien planets to take the war to the Bugs. They are a ruthless enemy with only one mission: Survival of their species no matter what the cost..."
}
]
""";
}




Movie.cs


public class Movie
{
    public string name { get; set; }
    public int year { get; set; }
    public int runtime { get; set; }
    public List<string> categories { get; set; }
    public string releasedate { get; set; }
    public string director { get; set; }
    public List<string> writer { get; set; }
    public List<string> actors { get; set; }
    public string storyline { get; set; }
}


Let's suppose the method GetActorsByMovieTitle is called many times or takes a lot of time to calculate. We want to cache it, to memoize it. It will be cached in a simple manner using memoize. This will short term cache the results, if we would like to persist the memoized results for long duration, we would use some other caching service such as database or Redis cache. The caching will function in sequential calls inside the same scope, it could be scoped as a singleton and long term cached inside memory for example. So here is how we can do the memoization shown below.

FunctionalExtensions.cs


public static Func<T1, TOut> Memoize<T1, TOut>(this Func<T1, TOut> @this, Func<T1, string> keyGenerator)
	{
		var dict = new Dictionary<string, TOut>();
		return x =>
		{
			string key = keyGenerator(x);
			if (!dict.ContainsKey(key))
			{
				dict.Add(key, @this(x));
			}
			return dict[key];
		};
	}
	public static Func<T1, T2, TOut> Memoize<T1, T2, TOut>(this Func<T1, T2, TOut> @this, Func<T1, T2, string> keyGenerator)
	{
		var dict = new Dictionary<string, TOut>();
		return (x,y) =>
		{
			string key = keyGenerator(x,y);
			if (!dict.ContainsKey(key))
			{
				dict.Add(key, @this(x,y));
			}
			return dict[key];
		};
	}
	public static Func<T1, T2, T3, TOut> Memoize<T1, T2, T3, TOut>(this Func<T1, T2, T3, TOut> @this, Func<T1, T2, T3, string> keyGenerator)
	{
		var dict = new Dictionary<string, TOut>();
		return (x, y, z) =>
		{
			string key = keyGenerator(x, y,z);
			if (!dict.ContainsKey(key))
			{
				dict.Add(key, @this(x, y, z));
			}
			return dict[key];
		};
	}
	public static Func<T1, T2, T3, T4, TOut> Memoize<T1, T2, T3, T4, TOut>(this Func<T1, T2, T3, T4, TOut> @this, Func<T1, T2, T3, T4, string> keyGenerator)
	{
		var dict = new Dictionary<string, TOut>();
		return (x, y, z, w) =>
		{
			string key = keyGenerator(x, y, z, w);
			if (!dict.ContainsKey(key))
			{
				dict.Add(key, @this(x, y, z, w));
			}
			return dict[key];
		};
	}


As we see above, we use a dictionary inside the memoize overloads and the way generics works, a dictionary will live inside each overloaded method accepting a different count of generic type parameters. We also provide a keyGenerator method that must be supplied to specify how we build up a unique key that we decide how we shall key each results from the given set of parameter(s). Note that we return here a function result, that is a func, that returns TOut and accepts the specified parameters in each overload. T1 or T1,T2 or T1,T2,T3 or T1,T2,T3,T4 and so on. Expanding the methods above to for example 16 parameters would be fairly easy, the code above shows how we can add support for more and more parameters. I believe you should avoid methods with more than 7 parameters,
but the code above should be clear. We return a func and we also accept also a func which returns TOut and same amount of parameters of same types T1,.. in each overload. Okay, next up an example how we can use this memoize function in the main method.

Program.cs


void Main()
{
    var movieStore = new MovieStore();
    
    //string actors = movieStore.GetActorsByMovieTitle("Starship troopers");
    //actors.Dump("Starship Troopers - Actors");
    //
    //Demo of memoized function
    
    var GetActorsByMovieTitle = ((string movieTitle) => movieStore.GetActorsByMovieTitle(movieTitle));
    var GetActorsByMovieTitleM = GetActorsByMovieTitle.Memoize(x => x);
    
    var starShipTroopersActors1 = GetActorsByMovieTitleM("Starship troopers");
    starShipTroopersActors1.Dump("Starship troopers - Call to method #1 time");
    var starShipTroopersActors2 = GetActorsByMovieTitleM("Starship troopers");
    starShipTroopersActors2.Dump("Starship troopers - Call to method #2 time");
    var starShipTroopersActors3 = GetActorsByMovieTitleM("Starship troopers");
    starShipTroopersActors3.Dump("Starship troopers - Call to method #3 time");
}


Note that in the test case above we send in one parameter T1 of type string, which is a movie title and we declare a func variable first using a lambda. We have to do the memoization in two declarations here and we use the convention that we suffix the memoized function with 'M' for 'Memoize'

Program.cs


void Main()
{
    var movieStore = new MovieStore();    
    var GetActorsByMovieTitle = ((string movieTitle) => movieStore.GetActorsByMovieTitle(movieTitle));
    var GetActorsByMovieTitleM = GetActorsByMovieTitle.Memoize(x => x);

The code has added a Console.WriteLine in the method which is memoized to check how many times the method is actually called or the cached result is returned instead. A run in Linqpad 7 is shown in screenshot below, showing that the output is cached correct. Note that if we wanted a thread implementation, we could instead use ConcurrentDictionary for example. The following methods show how we can do this. We exchanged Dictionary with ConcurrentDictionary and exchanged Add with TryAdd method of ConcurrentDictionary.

Program.cs


	public static Func<T1, TOut> MemoizeV2<T1, TOut>(this Func<T1, TOut> @this, Func<T1, string> keyGenerator)
	{
		var dict = new ConcurrentDictionary<string, TOut>();
		return x =>
		{
			string key = keyGenerator(x);
			if (!dict.ContainsKey(key))
			{
				dict.TryAdd(key, @this(x));
			}
			return dict[key];
		};
	}
	public static Func<T1, T2, TOut> MemoizeV2<T1, T2, TOut>(this Func<T1, T2, TOut> @this, Func<T1, T2, string> keyGenerator)
	{
		var dict = new ConcurrentDictionary<string, TOut>();
		return (x, y) =>
		{
			string key = keyGenerator(x, y);
			if (!dict.ContainsKey(key))
			{
				dict.TryAdd(key, @this(x, y));
			}
			return dict[key];
		};
	}
	public static Func<T1, T2, T3, TOut> MemoizeV2<T1, T2, T3, TOut>(this Func<T1, T2, T3, TOut> @this, Func<T1, T2, T3, string> keyGenerator)
	{
		var dict = new ConcurrentDictionary<string, TOut>();
		return (x, y, z) =>
		{
			string key = keyGenerator(x, y, z);
			if (!dict.ContainsKey(key))
			{
				dict.TryAdd(key, @this(x, y, z));
			}
			return dict[key];
		};
	}
	public static Func<T1, T2, T3, T4, TOut> MemoizeV2<T1, T2, T3, T4, TOut>(this Func<T1, T2, T3, T4, TOut> @this, Func<T1, T2, T3, T4, string> keyGenerator)
	{
		var dict = new ConcurrentDictionary<string, TOut>();
		return (x, y, z, w) =>
		{
			string key = keyGenerator(x, y, z, w);
			if (!dict.ContainsKey(key))
			{
				dict.TryAdd(key, @this(x, y, z, w));
			}
			return dict[key];
		};
	}


Hopefully, memoize or the process of memoization should be clearer now. It is a call based caching technique used preferably for pure functions / methods that has the same or equivalent result given a set of input parameter(s) and we memoize the function / method and cache the results. When used inside e.g. a singleton, we can cache longer time in memory and achieve performance boosts. You could do the same of course using a static variable, but the memoize technique is more generic purpose and is a pattern that is used in many programming languages. F# usually got way better support for functional programming than C#, but actually lacks a built in memoization functionality. Other languages do support memoization built in, such as in Python and LISP. The following screen shot shows a run of memoization above, I used ConcurrentDictionary when I tested.

Tuesday, 2 July 2024

Maybe Monad in C# - Guarding against nulls

This article will look more at the Maybe monad in C#. It is used to guard against nulls and is one of the most known monads in functional programming. The Maybe monad is also called Option or Optional many places. The screenshot below shows the allowed state transitions.

First off, records will be used since they support immutability out of the box.

Maybe.cs


public abstract record Maybe<T>();
public record Nothing<T> : Maybe<T>;
public record UnhandledNothing<T> : Nothing<T>;
public record Something<T>(T Value) : Maybe<T>;
public record Error<T>(Exception CapturedError) : Maybe<T>;
public record UnhandledError<T>(Exception CapturedError) : Error<T>(CapturedError);


As we see, the base record Maybe of T is abstract and can be one of several subtypes. Nothing means there are no value contained inside the 'container' or monad. Something of T means a value is inside the 'container'. This value can be null, but the container wrapping this value is not null, hence by sticking to monads such as Maybe we avoid null issues for the code that lives outside using the container, accessing the container. Maybe container here is no magic bullet, but it will make it easier to avoid null issues in your code in the parts where it is used. Let's next look at extension methods for the Maybe types defined in the records above.

MaybeExtensions.cs


public static class MaybeExtensions
{

	/// <summary>ToMayBe operates as a Return function in functional programming, lifting a normal value into a monadic value</summary>
	public static Maybe<T> ToMaybe<T>(this T @this)
	{		
		if (!EqualityComparer<T>.Default.Equals(@this, default))
		{
			return new Something<T>(@this);
		}
		else if (@this != null && Nullable.GetUnderlyingType(@this.GetType()) == null && @this.GetType().IsPrimitive){
			//primitive types that are not nullable and has got a default value are to be considered to have contents and have Something<T>, for example int value 0 or bool value false
			return new Something<T>(@this);
		}		
		return new Nothing<T>();
	}
	
	/// <summary>TryGetValue is similar to Reduce method in functional programming, but signal a boolean flag if the retrieval of value was successful</summary>
	public static bool TryGetValue<T>(this Maybe<T> @this, out T value){
	 	value = @this switch {
			Something<T> s => s.Value,
			_ => default(T)			
		};
		return @this is Something<T>;
	}
	
    //<summary>Call method Bind first to get correct behavior;/summary>
	public static Maybe<T> OnSomething<T>(this Maybe<T> @this, Action<T> actionOnSomething){
		if (@this is Something<T> s){
			actionOnSomething(s.Value);
		}		
		return @this;
	}
	
    //<summary>Call method Bind first to get correct behavior;/summary>
	public static Maybe<T> OnNothing<T>(this Maybe<T> @this, Action actionOnNothing){
		if (@this is UnhandledNothing<T>){
			actionOnNothing();
			return new Nothing<T>(); //switch from UnhandledNothing<T> to Nothing<T>
		}
		return @this;
	}
	
    //<summary>Call method Bind first to get correct behavior;/summary>
	public static Maybe<T> OnError<T>(this Maybe<T> @this, Action<Exception> actionOnError){
		if (@this is UnhandledError<T> e){
			actionOnError(e.CapturedError);
			return new Error<T>(e.CapturedError); //switch from UnhandledError<T> to Error<T>
		}
		return @this;
	}
		
	/// <summary>Bind is similar to Map in functional programming, it applies a function to update and allow switching from TIn to TOut data types, possibly different types</summary>
	public static Maybe<TOut> Bind<TIn, TOut>(this Maybe<TIn> @this, Func<TIn, TOut> f){
		try
		{
			Maybe<TOut> updatedMaybe = @this switch {
			    null => new Error<TOut>(new Exception("Object input is null")),
				Something<TIn> s when !EqualityComparer<TIn>.Default.Equals(s.Value, default) => new Something<TOut>(f(s.Value)),
				Something<TIn> s when @this.GetType().GetGenericArguments().First().IsPrimitive && Nullable.GetUnderlyingType(@this.GetType()) == null => new Something<TOut>(f(s.Value)),
				Something<TIn> _ => new UnhandledNothing<TOut>(),
				UnhandledNothing<TIn> _ => new UnhandledNothing<TOut>(),
				Nothing<TIn> _ => new Nothing<TOut>(),
				UnhandledError<TIn> u => new UnhandledError<TOut>(u.CapturedError),
				Error<TIn> e => new Error<TOut>(e.CapturedError),
				_ => new Error<TOut>(new Exception($"Got a subtype of Maybe<T>, which is not supported: {@this?.GetType().Name}"))				
			};
			return updatedMaybe;			
		}
		catch (Exception ex)
		{			
			return new UnhandledError<TOut>(ex);
		}		
	}
		
}


The extension methods above handle the basics for the Maybe of T monad.
  • We transform from a value of T to the Maybe of T using the method ToMaybe, this method is called Return many places. I think the name ToMaybe is more intuitive for more developers in C#.
  • The method TryGetValue is called Reduce many places and extracts the value of Maybe of T if it is available and returns a boolean value if the Maybe of T actually contains a value. If Maybe of T is not the type Something of T it does not hold a value, so knowing this is useful after retrieving the value.
  • The method Bind is sometimes called Map and allows updates of the value inside the Maybe of T and also perform transitions of sub types of Maybe of T and uses pattern matching in C#. Bind both maps and also performs the overall flow of state via controlling the sub type of Maybe of T as shown in the pattern specified inside Bind
  • The methods OnError, OnNothing, OnSomething are callbacks to do logic when retrieving values of these types that inherit from Maybe of T. Make sure you call the method Bind first
If you know more methods a Maybe of T monad should support, please let me know. A more detailed example of a Maybe of T monad can be seen in this implementation, note that it is called Option of T instead.

https://github.com/nlkl/Optional/blob/master/src/Optional/Option_Maybe.cs

The benefit with the code in this article is that it is shorter and that it uses records in C#. Since it is shorter, it should be easier to adapt and adjust to the needs. Demo code is next. Consider this record:

Car.cs


public record Car(string Model, string Make, bool? IsFourWheelDrive = null, string? Color = null);




Program.cs


void Main()
{
	var something = new Something<string>("hullo");
	something = something with { Value = null };
	
	var somethingnull = EqualityComparer<string>.Default.Equals(something.Value, default);
	
	int? nullableNumber = 1;
	Maybe<int?> maybe = nullableNumber.ToMaybe();

	var volvo = new Car("240 GL", "Volvo");

	bool isValueRetrieved = volvo.ToMaybe()
		.Bind(car => car with { IsFourWheelDrive = false })
		.Bind(car => car with { Color = "Blue" })
		.TryGetValue(out Car updatedVolvo);

	new { updatedVolvo, isValueRetrieved }.Dump("Updated Volvo");
	
	Maybe<string> noCar = new Something<string>(null)
							.Bind(x => x)
							.OnNothing(() => Console.WriteLine("The Car is nothing (null)!"));
	noCar.Dump("No car");
		
	maybe.Dump("Maybe");
	
	something.Dump();
	
}



Output from Linqpad shows the result from running the demo code. Note that since a record Car was used in the demo code, inside the Bind calls it was necessary to use the 'with' to mutate the record and create a new record
with the necessary adjustments. Also, it is important to call Bind before handlers OnNothing, OnError and OnSomething to work properly in the code, since the transition rules inside Bind should run first to check the correct subtype of Maybe of T. Many implementations of Option of T or Maybe of T that is, also implement operators for equality comparison of the value inside the Maybe of T if there is a defined value there (i.e. Something of T). The mentioned example on Github goes in detail on this and adds several methods for equality comparisons. Also, there are examples of support async methods such as BindAsync in Maybe of T implementations out there. Many of the building blocks of functional programming in C# are not standardized or built into the framework, so working out these building blocks yourself to tailour your source code needs is probably a good solution, if not going for libraries such as LanguageExt.

https://github.com/louthy/language-ext

Finally, notice that in the screenshot below, we have a Value of null for the variable something in the demo code. To get the proper sub type of Maybe of T, we should also call the Bind method. If we instead of :

  something.Dump();

Do this:

  something.Bind(x => x).Dump();

We get the correct subtype of Maybe of T, UnhandledNothing of T.

And if we also handle the UnhandledNothing of T, we get Nothing of T. 

something.Bind(x => x).OnNothing(() => Console.WriteLine("something is null")).Dump();

Sunday, 30 June 2024

Fork combinator revisited - supporting multiple part functions in C#

This article shows an example of a Fork combinator or 'monad' that will allow you to specify a join function that operates on all the part results and allow you to specify multiple part functions to be operated in sequence. First off, we define a simple map monad to map a value to another value of possibly other type. Then we define the fork combinator. The code below is very simple and short, it uses LINQ functionality to combine the results via the Select method, Linq is also functional so this is how we build up functional monads in C#, using Linq and Func and generics (and pattern matching and more).

Combinators.cs


public static class Combinators {
	
	public static TOut Map<TIn, TOut>(this TIn @this, Func<TIn, TOut> f) => f(@this);
	
	public static TOut Fork<TIn, TMiddle, TOut>(this TIn @this, Func<IEnumerable<TMiddle>, TOut> joinFunc,
		params Func<TIn, TMiddle>[] partFuncs) => partFuncs.Select(pf => pf(@this)).Map(joinFunc);
	
}


Let's look at a simple demo how to use this

Program.cs


public static class Program {
	
	string hello = "hhhhhheeeeeeeelllllllllllooooo";
	
	int sumOfLettersToLookFor = hello.Fork(results => (int)results.Sum(), 
				x => (double)x.Count(l => l == 'h'),
				x => (double) x.Count(l => l == 'e'),
				x => (double) x.Count(l => l == 'l'),
				x => (double) x.Count(l => l == 'o'));
	
	sumOfLettersToLookFor.Dump();
	
}


Functional programming has many of these monads that are very short and allows you to do combinations that would be lengthy and stateful in the procedural / object oriented way but elegant and short in the functional world. Finally a screenshot from Linqpad 7 showing the code above works : (A reference to Jerry Seinfeld to the right for those who know Seinfeld episodes)

Iterating a list using Span in C# - Benchmark

It is possible to iterate a list in C# using Span. This article shows a benchmark on that and how you can create a Span of a list. It is important to note that the Span will despite it is not doing any memory to memory copying and is performance optimized, it will not
detect changes done to the list after the span is created, therefore the iterations you perform should not be done on a list where you expect that the list is altered during the iteration using the Span. Let's first look at the Benchmark, BenchmarkDotNet will be used. The Github repo with the source code for the benchmark is here: https://github.com/toreaurstadboss/BenchmarkIterateCollections The program is started up using BenchmarkRunner
Program.cs


using BenchmarkDotNet.Running;

namespace BenmarkIterateCollections;


public class Program
{

    public static void Main(string[] args)
    {
        BenchmarkRunner.Run<IterateCollectionsBenchmark>();
        Console.WriteLine("Benchmark finished. Hit the any key to exit");
        Console.ReadKey();
    }
}



The benchmarks look like this, the base line is iterating an array.
BenchmarkIterateCollections.cs


using BenchmarkDotNet.Attributes;
using System.Runtime.InteropServices;

namespace BenmarkIterateCollections;

public class IterateCollectionsBenchmark
{

    private Random _rnd = new Random();
    private int[] _array = default!;

    private List<int> _list = default!;
    private const int ITEM_COUNT = 1001;

    [GlobalSetup]
    public void Init()
    {
        _array = Enumerable.Range(0, ITEM_COUNT).Select(_ => _rnd.Next(0, 1024)).ToArray();
        _list = _array.ToList();
    }

    [Benchmark(Description = "Iterate an array with foreach", Baseline = true)]
    public void IterateArray()
    {
        foreach (var item in _array)
        {
        }
    }

    [Benchmark(Description = "Iterate a list using a span with foreach")]
    public void IterateArrayUsingSpan()
    {
        Span<int> span = _array; //note : implicit operator used here to convert an array to a Span
        foreach (var item in span)
        {

        }
    }

    [Benchmark(Description = "Iterate a list with foreach")]
    public void IterateList()
    {
        foreach (var item in _list)
        {

        }
    }

    [Benchmark(Description = "Iterate a list using a span with foreach")]
    public void IterateListUsingSpan()
    {
        Span<int> span = CollectionsMarshal.AsSpan(_list);
        foreach (var item in span)
        {

        }
    }

}



Conclusions

Iterating an array using a span actually ran 6% faster than iterating an array without using the span. Iteating a list using a span ran some 50% faster than iterating the list using the span. The code above used the method CollectionsMarshal.AsSpan() in System.Runtime.InteropServices namespace - this has been available since .NET 5.x. Being aware of this method means we can iterate lists about 50% faster keeping in mind that changes to the list after the span was created from the list will not be shown in the span and we could show stale data if changes occur. You can also use this in .NET Framework using System.Memory Nuget package For large lists you know will not change, using spans to iterate could and should promise a large performance improvement. It is also possible to slice a portion of lists or arrays in case it is a case of iterating parts of a large list or array.

Wednesday, 26 June 2024

Functional programming - Guarding against nulls with the Option monad

Monads are "elementary individual substances". We can call them "building blocks" in Functional Programming (FP) and FP is built upon such building blocks to compose more complex logic. This article sums up using the

Option

monad, which is described many places on the net and is wellknown. This article is just a walkthrough of this monad, which creates a wrapper for a value that will aid against guarding against nulls. We will effectively abstract away null, and using extension methods we will allow setting up a processing chain where nulls are wrapped inside a value inside the Option objects. First off, define a generic class for this monad and in this case we will consider reference types or classes (this includes strings). Also, equality operators can be added to make it easier to do equality checks. The class will be called Option and we will define two methods called None and Some. The field called _content of type T will be returned inside a Option wrapper instance and None will only return Option object where _content is null.
  • None() signals that the payload or _content is null. None method returns an instance of Option of T where the wrapped value is null.
  • Some() signals that the payload or _content> is not null. some method returns an instance of Option of T where the wrapped value is not null.
  • Both the None() and Some() are of type Option<T> and can be used in a chained expression.
Some concepts of what the Option should support :
  • A map method that allows transforming from T to TResult (both can be same type) along the chained expressions
  • A reduce method that allows retrieving the value inside the Option object that is wrapped
  • Extension methods for going from an object to an Option wrapper of the object
  • Predicate based filter methods for peeling away Option objects where the wrapped object which satisfy the filter can be filtered



public class Option<T> : IEquatable<Option<T>> where T : class {
	
	private T? _content;
	
	private Option(){}

	public static Option<T> None() => new();
	public static Option<T> Some(T obj) => new Option<T> { _content = obj };

	public override bool Equals(object? obj)
	{
		return this.Equals(obj as Option<T>);
	}
	
	public Option<T> Where(Func<T, bool> predicate) => 
		_content is not null && predicate(_content) ? this : Option<T>.None();

	public Option<T> WhereNot(Func<T, bool> predicate) =>
		_content is not null && !predicate(_content) ? this : Option<T>.None();

	public bool Equals(Option<T>? other) => other is null ? false :
		_content?.Equals(other._content) ?? false;
		
	public static bool operator ==(Option<T>? a, Option<T>? b) => 
		a is null ? b is null : a.Equals(b);
		
	public static bool operator !=(Option<T>? a, Option<T>? b) => !(a == b);
		
	public override int GetHashCode()
	{
		return _content?.GetHashCode() ?? 0;
	}

	public Option<TResult> MapOptional<TResult>(Func<T, Option<TResult>> map) where TResult : class =>
		_content is not null ? map(_content) : Option<TResult>.None();

	public Option<TResult> Map<TResult>(Func<T, TResult> map) where TResult : class =>
		new Option<TResult>() { _content = _content is not null ? map(_content) : null };
	
    public T Reduce(T orElse) => _content ?? orElse;
	
	public T Reduce(Func<T> orElse) => _content ?? orElse();
	
} 


We also add some extension methods to help using the monad Option above containing the Some and None methods :

 
 
 public static class OptionExtensions {
	
	public static Option<T> ToOption<T>(this T? obj) where T : class => obj is not null ? Option<T>.Some(obj) : Option<T>.None();

	public static Option<T> Where<T>(this T? obj, Func<T, bool> predicate) where T : class => obj is not null && predicate(obj) ? Option<T>.Some(obj) : Option<T>.None();
	
	public static Option<T> WhereNot<T>(this T? obj, Func<T, bool> predicate) where T : class => obj is not null && !predicate(obj) ? Option<T>.Some(obj) : Option<T>.None();

}

 
Let's also add a ForEach extension method for easier usage in chained expressions of ienumerable collections.
 
 
 public static class IEnumerableExtensions {
	
	public static IEnumerable<T> ForEach<T>(this IEnumerable<T> items, Action<T> action){
		foreach (var item in items){
			action(item);
		}
		return items;
	}
	
}
 
 

The demo code then looks like below, where we iterate through an array of names and grab the initials of the name with some simple logic. Note usage of WhereNot predicate to avoid having to think about null strings further down the chain and also the usage of the Reduce method to get the value which the Option monad wraps.


void Main()
{
	string?[] authors = new[]{
	 "Johan Bojer",
	 null,
	 "Henrik Ibsen",
	 "Kristoffer Updahl",
	};
	
    authors.Select(x => GetInitial(x).Reduce(() => string.Empty)).ForEach(Console.WriteLine);
    
    //using ToOption() method
    authors.Select(a => GetInitial(a.ToOption())).ForEach(o => Console.WriteLine(o.Reduce(() => "?")));

	
	
}

//methods GetInitial below

Option<string> GetInitial(Option<string> name) => name.WhereNot(string.IsNullOrWhiteSpace).Map(s => s.TrimStart().Substring(0, 1).ToUpper());


The example in this article with retrieving initial (first letter) of some names is trivial. The real benefit of using the Option is to safeguard against nulls in lengthy chained calls or other scenarios where you want to always return something and decide how to indicate that we got something back and decide what an empty object will be, in the example the letter "?" signals a null based name.

Sunday, 16 June 2024

Adding synchronous and asynchronous foreach support to Range in C#

In C# 8, we got Range and Index. This allows you to construct a sequence of values. This article will look at diferent ways of adding foreach support in C# to Range. Let's first look at enumeration in the synchronous case. This requires us to add an enumerator to Range as an extension method. We define the following extension method GetEnumerator on Range to achieve this: This requires us to create a custom enumerator class that has these methods :
  • MoveNext
  • Current
The rest of the custom enumerator is happening via the logic in these methods and the constructor of this custom enumerator.


public static class RangeExtensions
{
	public static CustomRangeEnumerator GetEnumerator(this Range range) => new CustomRangeEnumerator(range);

	public class CustomRangeEnumerator
	{

		private int _current;
		private readonly int _end;
		private readonly int _start;
		private readonly bool _isDescending;

		public CustomRangeEnumerator(Range range)
		{
			if (range.End.IsFromEnd)
			{
				throw new NotSupportedException();
			}
			_end = range.End.Value;
			_start = range.Start.Value;
			_isDescending = _start > _end;
			_current = _isDescending ? range.Start.Value + 1 : range.Start.Value - 1;
		}

		public int Current => _current;

		public bool MoveNext()
		{
			if (_isDescending)
			{
				_current--;
				return _current >= _end;
			}
			else
			{
				_current++;
				return _current <= _end;
			}
		}
	}
}



The following code tests out this range enumeration:


Console.WriteLine($"Iterating range 10..1");
	foreach (var num in (10..1))
	{
		Console.WriteLine(num);
	}

	Console.WriteLine($"Iterating range 20..30");
	foreach (var num in 20..30)
	{
		Console.WriteLine(num);
	}


And we get the following output, showing we can enumerate a range both increasing and decreasing :


10
9
8
7
6
5
4
3
2
1

Iterating range 20..30
20
21
22
23
24
25
26
27
28
29
30


Let's look next how we can implement async support for iterations. I first started with having a go with IAsyncEnumerator. But I could not make the cancellation token work, since I could not pass it in. Instead I landed on a method GetItemsAsync that supports cancellation tokens being passed in and will respect cancellation. Here is the extension methods I created :



public static class RangeExtensions
{

	public static async IAsyncEnumerable<int> GetItemsAsync(this Range range, CancellationToken cancellationToken = default)
	{
		await foreach (var num in range)
		{
			if (cancellationToken.IsCancellationRequested){
				Console.WriteLine($"Iteration FAILED: Exiting the IAsyncEnumerable iteration as soon as possible since cancellation was requested at index: {num}");
			}
			cancellationToken.ThrowIfCancellationRequested();
			await Task.Delay(100);
			yield return num;
		}
	}
	
	// Existing asynchronous enumeration with delay and cancellation token
	public static CustomIntAsyncEnumerator GetAsyncEnumerator(this Range range)
		=> new CustomIntAsyncEnumerator(range);

	public class CustomIntAsyncEnumerator : IAsyncEnumerator<int>
	{
		private int _current;
		private readonly int _start;
		private readonly int _end;
		private readonly bool _isDescending;

		public CustomIntAsyncEnumerator(Range range)
		{
			_start = range.Start.Value;
			_end = range.End.Value;
			_isDescending = _start > _end;
			_current = _isDescending ? _start + 1 : _start - 1;
		}

		public int Current => _current;

		public async ValueTask<bool> MoveNextAsync()
		{
			// Wait for the specified delay time
			await Task.Delay(0);

			if (_isDescending)
			{
				if (_current > _end)
				{
					_current--;
					return true;
				}
			}
			else
			{
				if (_current < _end)
				{
					_current++;
					return true;
				}
			}

			return false;
		}

		public ValueTask DisposeAsync()
		{
			// Perform any necessary cleanup here
			return new ValueTask();
		}
	}
}



The code below tests the async enumeration. The benefit of the IAsyncEnumerator is that you can iterate without any hassles with cancellation token, but you should respect cancellation tokens. So the last method call calling the method GetItemsAsync do respect the cancellation token and also accepts such a token being passed in. I tested out the code in this article with Linqpad 7, and developed it using this small IDE.


    var rangeOneHundredDownTo90 = 100..90;

	Console.WriteLine($"Await foreach over variable {nameof(rangeOneHundredDownTo90)}");
	await foreach (var num in rangeOneHundredDownTo90)
	{
		Console.WriteLine(num);
	}

	Console.WriteLine("Async iteration from 140 to 180, waiting 500 ms between each iteration");

	var range = 140..180;

	var cts = new CancellationTokenSource();
	var token = cts.Token;

	await foreach (var num in range.GetItemsAsync(token))
	{
		if (num >= 170)
		{
			cts.Cancel();
		}
		Console.WriteLine(num);
	}



The screen shot below shows that the cancellation token was throwing a OperationCanceledException as was the intent of the demo code.

Saturday, 18 May 2024

Discriminated Union Part Two - The C# side of things

In this article , discriminated unions will be further looked into, continuing from the last article. It visited these topics using F#. The previous article showing the previous article focused on F# and discriminated unions is available here:

https://toreaurstad.blogspot.com/2024/05/discriminated-unions-part-one-f-side-of.html

In this article, C# will be used. As last noted, discriminated unions are a set of types that are allowed to be used. In F#, these types dont have to be in an inheritance chain, they can really be a mix of different types. In C# however, one has to use a base type for the union itself and declare this as abstract, i.e. a placeholder for our discriminated union, called DU from now in this article. C# is a mix of object oriented and functional programming language. It does not support discriminated unions as built-in constructs, such as F#. We must use object inheritance still, but pattern matching in C# with type testing. Lets first look at the POCOs that are included in this example, we must use a base class for our union. In F# we had this:


type Shape =
    | Rectangle of width : float * length : float
    | Circle of radius : float
    | Prism of width : float * depth:float * height : float
    | Cube of width : float


In C# we use an abstract record, since they possess immutability after construction has been made and are therefore a good match for functional programming (FP). Also, records offer a compact syntax which lets itself nice to FP. (we COULD use an abstract class too, but records are now available and lends themsevles better to FP since they are immutable after construction is finished). We could define this abstract record baseclass which will function as a Discriminated Union (DU) like:


public abstract record Shape;


However, keeping note of which types are allowed into the DU is easier if we nest our types. I have also included the methods on the Shape objects as static methods that uses pattern matching with type testing to define the bounds of our DU.



public abstract record Shape
{

	public record Rectangle(float Width, float Length) : Shape;
	public record Circle(float Radius) : Shape;
	public record Prism(float Width, float Depth, float Length) : Shape;
	public record Cube(float Width) : Shape;
	public record Torus(float LargeRadius, float SmallRadius) : Shape; //we will discriminate this shape, not include it in our supported calculations

	public static double CalcArea(Shape shape) => shape switch
	{
		Rectangle rect => rect.Width * rect.Length,
		Circle circ => Math.PI * Math.Pow(circ.Radius, 2),
		Prism prism => 2.0*(prism.Width*prism.Depth) + 2.0*(prism.Width+prism.Depth)*prism.Length,
		Cube cube => 6 * Math.Pow(cube.Width, 2),
		_ => throw new NotSupportedException($"Area calculation for this Shape: ${shape.GetType()}")
	};

	public static double CalcVolume(Shape shape) => shape switch
	{
		Prism prism => prism.Width * prism.Depth * prism.Length,
		Cube cube => Math.Pow(cube.Width, 3),
		_ => throw new NotSupportedException($"Volume calculation for this Shape: ${shape.GetType()}")
	};

};


Sample code of using this source code is shown below:


void Main()
{
	var torus = new Shape.Torus(LargeRadius: 7, SmallRadius: 3);
	//var torusArea = Shape.CalcArea(torus);

	var rect = new Shape.Rectangle(Width: 1.3f, Length: 10.0f);
	var circle = new Shape.Circle(Radius: 2.0f);
	var prism = new Shape.Prism(Width: 15, Depth: 5, Length: 7);
	var cube = new Shape.Cube(Width: 2.0f);

	var rectArea = Shape.CalcArea(rect);
	var circleArea = Shape.CalcArea(circle);
	var prismArea = Shape.CalcArea(prism);
	var cubeArea = Shape.CalcArea(cube);

	//var circleVolume = Shape.CalcVolume(circle);
	var prismVolume = Shape.CalcVolume(prism);
	var cubeVolume = Shape.CalcVolume(cube);
	//var rectVolume = Shape.CalcVolume(rect);

	Console.WriteLine("\nAREA CALCULATIONS:");
	Console.WriteLine($"Circle area: {circleArea:F2}");
	Console.WriteLine($"Prism area: {prismArea:F2}");
	Console.WriteLine($"Cube area: {cubeArea:F2}");
	Console.WriteLine($"Rectangle area: {rectArea:F2}");

	Console.WriteLine("\nVOLUME CALCULATIONS:");
	//Console.WriteLine( "Circle volume: %A", circleVolume);
	Console.WriteLine($"Prism volume: {prismVolume:F2}");
	Console.WriteLine($"Cube volume: {cubeVolume:F2}");
	//Console.WriteLine( "Rectangle volume: %A", rectVolume);
}


I have commented out some lines here, they will throw an UnsupportedException if one uncomments them running the code. The torus forexample lacks support for area and volume calculation by intent, it is not supported (yet). The calculations of the volume of a circle and a rectangle is not possible, since they are 2D geometric figures and not 3D, i.e. do not posess a volume. Output from running the program is shown below:

AREA CALCULATIONS:
Circle area: 12,57
Prism area: 430,00
Cube area: 24,00
Rectangle area: 13,00

VOLUME CALCULATIONS:
Prism volume: 525,00
Cube volume: 8,00

Conclusions F# vs C#

True support for DU is only available in F#, but we can get close to it using C#, inheritance, pattern matching with type checking. F# got much better support for it for now, but C# probably will catch up in a few years and also finally get support for it as a built-in construct. The syntax for DU in F# an C# is fairly similar, using records and pattern switching with type checking makes the code in C# not longer than in F#, but F# got direct support for DU, in C# we have to add additional code to support something that is a built-in functionality of F#. Listed on the page What's new in C# 13, DU has not made their way into the list, .NET 9 Preview SDK will be available probably in November this year (2024).

https://learn.microsoft.com/en-us/dotnet/csharp/whats-new/csharp-13

There are different approaches to writing DU in C# for now. Some go for the OneOf operator of functional programming, not presented further in this article. Probably discriminated unions will make their way in .NET 10 in November 2027, so there will still be a lot of waiting around for getting this feature into C#. For now, being aware what the buzz about DU is all about, my two articles on it hopefully made it a bit clearer. One disadvantage of this is that it’s not consistent like in F#. We have to manually manage which types we want to support in each method. However, this is done using inheritance in C#. At the same time, we need to adjust the inheritance hierarchy so that all types inherit from such a discriminated union (DU). If a type needs to be part of MULTIPLE different DUs, we face limitations in C# since we can only inherit from a specific type in the hierarchy. This is likely why many C# developers are requesting DU functionality. As of now, Microsoft’s language team seems to be leaning toward something called ENUM CLASSES. It appears that this feature will be included in .NET 10, which means it won’t be available until 2027

Further viewing/reading of the topic

There are proposals for better support of DU in C# is taking its form now in concrete propals. A proposal for Enum classes are available here, it could be the design choice C# language team lands on:

https://github.com/dotnet/csharplang/blob/main/proposals/discriminated-unions.md

Lead Designer Mads Torgersen comments around DU in C# in this video at 21:00 :

https://learn.microsoft.com/en-us/shows/ask-the-expert/ask-the-expert-whats-new-in-c-100

Saturday, 11 May 2024

Discriminated Unions Part One - The F# side of things

I decided to look more into what the discussion of Discriminated unions in C#, or their lack of it is all about. I will first look at the F# side of things. How can we create a discriminated union in F# ? And then I will look at how we can implement the F# program in C# in the next article for the topic Discriminated unions. In this article we will look at some F# code that shows how discriminated unions are built-in supported in F#. Discriminated unions are special containers that can hold different types. This is not supported in C# without adding some additional plumbing code and it is not considered true discriminated unions, although in C# we can get close to Discriminated unions. For the rest of the article, we will call discriminated unions for DU. Let's first declare a DU in F# that describes different types of geometric figures.


type Shape =
    | Rectangle of width : float * length : float
    | Circle of radius : float
    | Prism of width : float * depth:float * height : float
    | Cube of width : float


The '*' operator in F# means when it is used in type definitions above as a separator of the properties that each type got,
e.g. Rectangle of width : float * length : float means
that the type Rectangle got two properties, width of type float and length of the same type.

Let's add some methods to our F# program, calculating area and calculating volume. We also want our F# to be fault tolerant so either we get a result or we get an error, for example this additional DU which is also generic.

type Result<'T> =
    | Success of 'T
    | Error of string    

We also neeed a way to print errors if we want to not crash the program, say if want to calculate the volume of a circle or a rectangle, which is not supported since it is 2D figures.


let handleResult (result: Result<float>) =
    match result with
    | Success value -> printfn "%A" value
    | Error msg -> printfn "Error: %s" msg; () // Return NaN for error cases


To add some functionality to the discriminated unions we add the module below:


module ShapeOperations =
    let CalcArea(shape : Shape) : Result<float> =
        match shape with 
        | Rectangle (width, length) -> Success(width * length)
        | Circle (radius) -> Success(Math.PI * radius**2)
        | Prism (width, depth, height) -> (2.0*(width*depth) + 2.0*(width+depth)*height)
        | Cube (width) -> Success(6.0 * width * width)
        // | _ -> failwith "Area calculation is not supported"
    let CalcVolume(shape : Shape) : Result<float> = 
        match shape with 
        | Prism (width, height, depth) -> Success(width * height * depth)
        | Cube (width) -> Success(width**3)        
        | _ -> Error(sprintf "Volume calculation is not supported  for: %A" shape)


The rest of the code is shown below where we instantiate geometric figures and calculate the area and volume of them and output their values.

 
let rect = Rectangle(length = 1.3, width = 10.0)
let circle = Circle (2.0)
let prism = Prism(width = 15, depth = 5.0, height = 7.0)
let cube = Cube(3)

let rectArea = ShapeOperations.CalcArea rect 
let circleArea = ShapeOperations.CalcArea circle
let prismArea = ShapeOperations.CalcArea prism
let cubeArea = ShapeOperations.CalcArea cube

let circleVolume = handleResult (ShapeOperations.CalcVolume circle)
let prismVolume = ShapeOperations.CalcVolume prism
let cubeVolume = ShapeOperations.CalcVolume cube
let rectVolume = ShapeOperations.CalcVolume rect

printfn "\nAREA CALCULATIONS:"
printfn "Circle area: %A" circleArea
printfn "Prism area: %A" prismArea 
printfn "Cube area: %A" cubeArea 
printfn "Rectangle area %A" rectArea 

printfn "\nVOLUME CALCULATIONS:"
printfn "Circle volume: %A" circleVolume 
printfn "Prism volume: %A" prismVolume 
printfn "Cube volume: %A" cubeVolume 
printfn "Rectangle volume: %A" rectVolume          
                      

We get this output after running the program :


Error: Volume calculation is not supported  for: Circle 2.0

AREA CALCULATIONS:
Circle area: Success 12.56637061
Prism area: Success 430
Cube area: Success 18.0
Rectangle area Success 13.0

VOLUME CALCULATIONS:
Circle volume: ()
Prism volume: Success 525.0
Cube volume: Success 27.0
Rectangle volume: Error "Volume calculation is not supported  for: Rectangle (10.0, 1.3)"


As we can see, creating DUs in F# is easy, we use the '|' operator to define multiple types and we can create generic DUs too and match different types with functional expressions. In the next article we will look at the code shown here and test out if we can recreate it in C# using different constructs. C# has gotten more support of functional programming in 2020 and most likely it will involve records, pattern matching (newer switch based syntax) and extension methods.

Thursday, 9 May 2024

Azure Cognitive Synthesized Text To Speech with voice styles

Using Azure Cognitive Services, it is possible to translate text into other languages and also synthesize the text to speech. It is also possible to add voice effects such as style of the voice. This adds more realism by adding emotions to a synthesized voice. The voice is already trained by neural net training and adding voice style makes the synthesized speech even more realistic and multi-purpose. The Github repo for this is available here as .NET Maui Blazor client written with .NET 8 :

MultiLingual translator DEMO Github repo

Not all the voices supported in Azure Cognitive Services do support voice effects. An overview of which voices are shown here:

https://learn.microsoft.com/nb-no/azure/ai-services/speech-service/language-support?tabs=tts#voice-styles-and-roles

More and more synthetic voices in Azure Cognitive Services gets more and more voice styles which express emotions. For now, most of the voices are either english (en-US) or chinese (zh-CN) and a few other languages got some few voices supporting styles. This will most likely be improved into the future where these neural net trained voices are trained in voice styles or some generic voice style algorithm is achieved that can infer emotions on a generic level, although that still sounds a bit sci-fi.

Azure Cognitive Text-To-Speech Voices with support for emotions / voice styles


Voice Styles Roles
de-DE-ConradNeural1 cheerful Not supported
en-GB-SoniaNeural cheerful, sad Not supported
en-US-AriaNeural angry, chat, cheerful, customerservice, empathetic, excited, friendly, hopeful, narration-professional, newscast-casual, newscast-formal, sad, shouting, terrified, unfriendly, whispering Not supported
en-US-DavisNeural angry, chat, cheerful, excited, friendly, hopeful, sad, shouting, terrified, unfriendly, whispering Not supported
en-US-GuyNeural angry, cheerful, excited, friendly, hopeful, newscast, sad, shouting, terrified, unfriendly, whispering Not supported
en-US-JaneNeural angry, cheerful, excited, friendly, hopeful, sad, shouting, terrified, unfriendly, whispering Not supported
en-US-JasonNeural angry, cheerful, excited, friendly, hopeful, sad, shouting, terrified, unfriendly, whispering Not supported
en-US-JennyNeural angry, assistant, chat, cheerful, customerservice, excited, friendly, hopeful, newscast, sad, shouting, terrified, unfriendly, whispering Not supported
en-US-NancyNeural angry, cheerful, excited, friendly, hopeful, sad, shouting, terrified, unfriendly, whispering Not supported
en-US-SaraNeural angry, cheerful, excited, friendly, hopeful, sad, shouting, terrified, unfriendly, whispering Not supported
en-US-TonyNeural angry, cheerful, excited, friendly, hopeful, sad, shouting, terrified, unfriendly, whispering Not supported
es-MX-JorgeNeural chat, cheerful Not supported
fr-FR-DeniseNeural cheerful, sad Not supported
fr-FR-HenriNeural cheerful, sad Not supported
it-IT-IsabellaNeural chat, cheerful Not supported
ja-JP-NanamiNeural chat, cheerful, customerservice Not supported
pt-BR-FranciscaNeural calm Not supported
zh-CN-XiaohanNeural affectionate, angry, calm, cheerful, disgruntled, embarrassed, fearful, gentle, sad, serious Not supported
zh-CN-XiaomengNeural chat Not supported
zh-CN-XiaomoNeural affectionate, angry, calm, cheerful, depressed, disgruntled, embarrassed, envious, fearful, gentle, sad, serious Boy, Girl, OlderAdultFemale, OlderAdultMale, SeniorFemale, SeniorMale, YoungAdultFemale, YoungAdultMale
zh-CN-XiaoruiNeural angry, calm, fearful, sad Not supported
zh-CN-XiaoshuangNeural chat Not supported
zh-CN-XiaoxiaoNeural affectionate, angry, assistant, calm, chat, chat-casual, cheerful, customerservice, disgruntled, fearful, friendly, gentle, lyrical, newscast, poetry-reading, sad, serious, sorry, whisper Not supported
zh-CN-XiaoyiNeural affectionate, angry, cheerful, disgruntled, embarrassed, fearful, gentle, sad, serious Not supported
zh-CN-XiaozhenNeural angry, cheerful, disgruntled, fearful, sad, serious Not supported
zh-CN-YunfengNeural angry, cheerful, depressed, disgruntled, fearful, sad, serious Not supported
zh-CN-YunhaoNeural2 advertisement-upbeat Not supported
zh-CN-YunjianNeural3,4 angry, cheerful, depressed, disgruntled, documentary-narration, narration-relaxed, sad, serious, sports-commentary, sports-commentary-excited Not supported
zh-CN-YunxiaNeural angry, calm, cheerful, fearful, sad Not supported
zh-CN-YunxiNeural angry, assistant, chat, cheerful, depressed, disgruntled, embarrassed, fearful, narration-relaxed, newscast, sad, serious Boy, Narrator, YoungAdultMale
zh-CN-YunyangNeural customerservice, narration-professional, newscast-casual Not supported
zh-CN-YunyeNeural angry, calm, cheerful, disgruntled, embarrassed, fearful, sad, serious Boy, Girl, OlderAdultFemale, OlderAdultMale, SeniorFemale, SeniorMale, YoungAdultFemale, YoungAdultMale
zh-CN-YunzeNeural angry, calm, cheerful, depressed, disgruntled, documentary-narration, fearful, sad, serious OlderAdultMale, SeniorMale

Screenshot from the DEMO showing its user interface. You enter the text to translate at the top and the language of the text is detected using Azure Cognitive Services text detection functionality. And you can then select which language to translate the text into. It will call a REST call to Azure Cognitive Services to translate the text. And it is also possible to hear the speech of the text. Now, it is also added to add voice style. Use the table shown above to select a voice actor that supports a voice style you want to test. As noted, voice styles are still limited to a few languages and voice actors supporting emotions or voice styles. You will hear the voice from the voice actor in a normal mood or voice style if additional emotions or voice styles are not supported.
Let's look at some code for this DEMO too. You can study the Github repo and clone it to test it out yourself. The TextToSpeechUtil class handles much of the logic of creating voice from text input and also create the SSML-XML contents and performt the REST api call to create the voice file. Note that SSML mentioned here, is the Speech Synthesis Markup Language (SSML). The SSML standard is documented here on MSDN, it is a standard adopted by others too including Google.

https://learn.microsoft.com/en-us/azure/ai-services/speech-service/speech-synthesis-markup



using Microsoft.Extensions.Configuration;
using MultiLingual.Translator.Lib.Models;
using System;
using System.Security;
using System.Text;
using System.Xml.Linq;
using static System.Runtime.InteropServices.JavaScript.JSType;

namespace MultiLingual.Translator.Lib
{
    public class TextToSpeechUtil : ITextToSpeechUtil
    {

        public TextToSpeechUtil(IConfiguration configuration)
        {
            _configuration = configuration;
        }

        public async Task<TextToSpeechResult> GetSpeechFromText(string text, string language, TextToSpeechLanguage[] actorVoices, 
            string? preferredVoiceActorId, string? preferredVoiceStyle)
        {
            var result = new TextToSpeechResult();

            result.Transcript = GetSpeechTextXml(text, language, actorVoices, preferredVoiceActorId, preferredVoiceStyle, result);
            result.ContentType = _configuration[TextToSpeechSpeechContentType];
            result.OutputFormat = _configuration[TextToSpeechSpeechXMicrosoftOutputFormat];
            result.UserAgent = _configuration[TextToSpeechSpeechUserAgent];
            result.AvailableVoiceActorIds = ResolveAvailableActorVoiceIds(language, actorVoices);
            result.LanguageCode = language;

            string? token = await GetUpdatedToken();

            HttpClient httpClient = GetTextToSpeechWebClient(token);

            string ttsEndpointUrl = _configuration[TextToSpeechSpeechEndpoint];
            var response = await httpClient.PostAsync(ttsEndpointUrl, new StringContent(result.Transcript, Encoding.UTF8, result.ContentType));

            using (var memStream = new MemoryStream()) {
                var responseStream = await response.Content.ReadAsStreamAsync();
                responseStream.CopyTo(memStream);
                result.VoiceData = memStream.ToArray();
            }

            return result;
        }

        private async Task<string?> GetUpdatedToken()
        {
            string? token = _token?.ToNormalString();
            if (_lastTimeTokenFetched == null || DateTime.Now.Subtract(_lastTimeTokenFetched.Value).Minutes > 8)
            {
                token = await GetIssuedToken();
            }

            return token;
        }

        private HttpClient GetTextToSpeechWebClient(string? token)
        {
            var httpClient = new HttpClient();
            httpClient.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", token);
            httpClient.DefaultRequestHeaders.Add("X-Microsoft-OutputFormat", _configuration[TextToSpeechSpeechXMicrosoftOutputFormat]);
            httpClient.DefaultRequestHeaders.Add("User-Agent", _configuration[TextToSpeechSpeechUserAgent]);
            return httpClient;
        }
       
        public string GetSpeechTextXml(string text, string language, TextToSpeechLanguage[] actorVoices, string? preferredVoiceActorId,
              string? preferredVoiceStyle, TextToSpeechResult result)
        {
            result.VoiceActorId = ResolveVoiceActorId(language, preferredVoiceActorId, actorVoices);
            string speechXml = $@"
            <speak version='1.0' xml:lang='en-US' xmlns:mstts='https://www.w3.org/2001/mstts'>
                <voice xml:gender='Male' name='Microsoft Server Speech Text to Speech Voice {result.VoiceActorId}'>
                    <prosody rate='1'>{text}</prosody>
                </voice>
            </speak>";

            speechXml = AddVoiceStyleEffectIfDesired(preferredVoiceStyle, speechXml);

            return speechXml;
        }

        /// <summary>
        /// Adds voice style / expression to the SSML markup for the voice
        /// </summary>
        private static string AddVoiceStyleEffectIfDesired(string? preferredVoiceStyle, string speechXml)
        {
            if (!string.IsNullOrWhiteSpace(preferredVoiceStyle) && preferredVoiceStyle != "normal-neutral")
            {
                var voiceDoc = XDocument.Parse(speechXml); //https://learn.microsoft.com/nb-no/azure/ai-services/speech-service/speech-synthesis-markup-voice#use-speaking-styles-and-roles

                XElement? prosody = voiceDoc.Descendants("prosody").FirstOrDefault();
                if (prosody?.Value != null)
                {
                    // Create the <mstts:express-as> element, for now skip the ':' letter and replace at the end

                    var expressedAsWrappedElement = new XElement("msttsexpress-as",
                        new XAttribute("style", preferredVoiceStyle));
                    expressedAsWrappedElement.Value = prosody!.Value;
                    prosody?.ReplaceWith(expressedAsWrappedElement);
                    speechXml = voiceDoc.ToString().Replace(@"msttsexpress-as", "mstts:express-as");
                }
            }

            return speechXml;
        }

        private List<string> ResolveAvailableActorVoiceIds(string language, TextToSpeechLanguage[] actorVoices)
        {
            if (actorVoices?.Any() == true)
            {
                var voiceActorIds = actorVoices.Where(v => v.LanguageKey == language || v.LanguageKey.Split("-")[0] == language).SelectMany(v => v.VoiceActors).Select(v => v.VoiceId).ToList();
                return voiceActorIds;
            }
            return new List<string>();
        }

        private string ResolveVoiceActorId(string language, string? preferredVoiceActorId, TextToSpeechLanguage[] actorVoices)
        {
            string actorVoiceId = "(en-AU, NatashaNeural)"; //default to a select voice actor id 
            if (actorVoices?.Any() == true)
            {
                var voiceActorsForLanguage = actorVoices.Where(v => v.LanguageKey == language || v.LanguageKey.Split("-")[0] == language).SelectMany(v => v.VoiceActors).Select(v => v.VoiceId).ToList();
                if (voiceActorsForLanguage != null)
                {
                    if (voiceActorsForLanguage.Any() == true)
                    {
                        var resolvedPreferredVoiceActorId = voiceActorsForLanguage.FirstOrDefault(v => v == preferredVoiceActorId);
                        if (!string.IsNullOrWhiteSpace(resolvedPreferredVoiceActorId))
                        {
                            return resolvedPreferredVoiceActorId!;
                        }
                        actorVoiceId = voiceActorsForLanguage.First();
                    }
                }
            }
            return actorVoiceId;
        }

        private async Task<string> GetIssuedToken()
        {
            var httpClient = new HttpClient();
            string? textToSpeechSubscriptionKey = Environment.GetEnvironmentVariable("AZURE_TEXT_SPEECH_SUBSCRIPTION_KEY", EnvironmentVariableTarget.Machine);
            httpClient.DefaultRequestHeaders.Add(OcpApiSubscriptionKeyHeaderName, textToSpeechSubscriptionKey);
            string tokenEndpointUrl = _configuration[TextToSpeechIssueTokenEndpoint];
            var response = await httpClient.PostAsync(tokenEndpointUrl, new StringContent("{}"));
            _token = (await response.Content.ReadAsStringAsync()).ToSecureString();
            _lastTimeTokenFetched = DateTime.Now;
            return _token.ToNormalString();
        }

        public async Task<List<string>> GetVoiceStyles()
        {
            var voiceStyles = new List<string>
            {
                "normal-neutral",
                "advertisement_upbeat",
                "affectionate",
                "angry",
                "assistant",
                "calm",
                "chat",
                "cheerful",
                "customerservice",
                "depressed",
                "disgruntled",
                "documentary-narration",
                "embarrassed",
                "empathetic",
                "envious",
                "excited",
                "fearful",
                "friendly",
                "gentle",
                "hopeful",
                "lyrical",
                "narration-professional",
                "narration-relaxed",
                "newscast",
                "newscast-casual",
                "newscast-formal",
                "poetry-reading",
                "sad",
                "serious",
                "shouting",
                "sports_commentary",
                "sports_commentary_excited",
                "whispering",
                "terrified",
                "unfriendly"
            };
            return await Task.FromResult(voiceStyles);
        }

        private const string OcpApiSubscriptionKeyHeaderName = "Ocp-Apim-Subscription-Key";
        private const string TextToSpeechIssueTokenEndpoint = "TextToSpeechIssueTokenEndpoint";
        private const string TextToSpeechSpeechEndpoint = "TextToSpeechSpeechEndpoint";        
        private const string TextToSpeechSpeechContentType = "TextToSpeechSpeechContentType";
        private const string TextToSpeechSpeechUserAgent = "TextToSpeechSpeechUserAgent";
        private const string TextToSpeechSpeechXMicrosoftOutputFormat = "TextToSpeechSpeechXMicrosoftOutputFormat";

        private readonly IConfiguration _configuration;

        private DateTime? _lastTimeTokenFetched = null;
        private SecureString _token = null;

    }
}

 
 

The REST call to generate the voice file is using following set up: TTS endpoint url: https://norwayeast.tts.speech.microsoft.com/cognitiveservices/v1 The transcript (text to translate into speech) is the following in my test as a SSML-XML document:


<speak version="1.0" xml:lang="en-US" xmlns:mstts="https://www.w3.org/2001/mstts">
  <voice xml:gender="Male" name="Microsoft Server Speech Text to Speech Voice (en-US, JaneNeural)">
    <mstts:express-as style="angry">I listen to Eurovision and cheer for Norway</mstts:express-as>
  </voice>
</speak>


The SSML also contains an extension called mstts extension language that adds features to SSML such as the express-as set to a voice style or emotion of "angry". Not all emotions or voice styles are supported by every voice actor in Azure Cognitive Services. But this is a list of the voice styles that could be supported, it varies which voice actor you choose (and inherently which language).
  • "normal-neutral"
  • "advertisement_upbeat"
  • "affectionate"
  • "angry"
  • "assistant"
  • "calm"
  • "chat"
  • "cheerful"
  • "customerservice"
  • "depressed"
  • "disgruntled"
  • "documentary-narration"
  • "embarrassed"
  • "empathetic"
  • "envious"
  • "excited"
  • "fearful"
  • "friendly"
  • "gentle"
  • "hopeful"
  • "lyrical"
  • "narration-professional"
  • "narration-relaxed"
  • "newscast"
  • "newscast-casual"
  • "newscast-formal"
  • "poetry-reading"
  • "sad"
  • "serious"
  • "shouting"
  • "sports_commentary"
  • "sports_commentary_excited"
  • "whispering"
  • "terrified"
  • "unfriendly
Microsoft has come a long way from the early work with SAPI - Microsoft Speech API with Microsoft SAM around 2000. The realism of synthetic voices more than 20 years ago were rather crude and robotic. Nowaydays, voice actors provided by Azure Cloud computing platform as shown here are neural net trained and very realistic based upon training from real voice actors and now more and more voice actor voices support emotions or voice styles. The usages of this can be diverse. Making use of text synthesis can serve in automated answering services and apps in diverse fields such as healthcare and public services or education and more. Making this demo has been fun for me and it can be used to learn languages and with the voice functionality you can train on not only the translation but also pronounciation.

Monday, 22 April 2024

Pii - Detecting Personally Identifiable Information using Azure Cognitive Services

This article will look at detecting Person Identifiable Information (Pii) using Azure Cognitive Services. I have created a demo using .NET Maui Blazor has been created and the Github repo is here:
https://github.com/toreaurstadboss/PiiDetectionDemo

Person Identifiable Information (Pii) is desired to detect and also redact, that is using censorship or obscuring Pii to prepare documents for publication. The Pii feature in Azure Cognitive Services is a part of the Language resource service. A quickstart for using Pii is available here:
https://learn.microsoft.com/en-us/azure/ai-services/language-service/personally-identifiable-information/quickstart?pivots=programming-language-csharp

After creating the Language resource, look up the keys and endpoints for you service. Using Azure CLI inside Cloud shell, you can enter this command to find the keys, in Azure many services has got two keys you can exchange with new keys through regeneration:

az cognitiveservices account keys list --resource-group SomeAzureResourceGroup --name SomeAccountAzureCognitiveServices
This is how you can query after endpoint of language resource using Azure CLI : az cognitiveservices account show --query "properties.endpoint" --resource-group SomeAzureResourceGroup --name SomeAccountAzureCognitiveServices
Next, the demo of this article. Connecting to the Pii Removal Text Analytics is possible using this Nuget package (REST calls can also be done manually): - Azure.AI.TextAnalytics version 5.3.0 Here is the other Nugets of my Demo included from the .csproj file :

PiiDetectionDemo.csproj


  <ItemGroup>
        <PackageReference Include="Azure.AI.TextAnalytics" Version="5.3.0" />
        <PackageReference Include="Microsoft.Maui.Controls" Version="$(MauiVersion)" />
        <PackageReference Include="Microsoft.Maui.Controls.Compatibility" Version="$(MauiVersion)" />
        <PackageReference Include="Microsoft.AspNetCore.Components.WebView.Maui" Version="$(MauiVersion)" />
        <PackageReference Include="Microsoft.Extensions.Logging.Debug" Version="8.0.0" />
    </ItemGroup>


A service using this Pii removal feature is simply making use of a TextAnalyticsClient and method RecognizePiiEntitiesAsync.

PiiRemovalTextClientService.cs IPiiRemovalTextClientService.cs



using Azure;
using Azure.AI.TextAnalytics;

namespace PiiDetectionDemo.Util
{
    public interface IPiiRemovalTextAnalyticsClientService
    {
        Task<Response<PiiEntityCollection>> RecognizePiiEntitiesAsync(string? document, string? language);
    }
}


namespace PiiDetectionDemo.Util
{
    public class PiiRemovalTextAnalyticsClientService : IPiiRemovalTextAnalyticsClientService
    {

        private TextAnalyticsClient _client;

        public PiiRemovalTextAnalyticsClientService()
        {
            var azureEndpoint = Environment.GetEnvironmentVariable("AZURE_COGNITIVE_SERVICE_ENDPOINT");
            var azureKey = Environment.GetEnvironmentVariable("AZURE_COGNITIVE_SERVICE_KEY");

            if (string.IsNullOrWhiteSpace(azureEndpoint))
            {
                throw new ArgumentNullException(nameof(azureEndpoint), "Missing system environment variable: AZURE_COGNITIVE_SERVICE_ENDPOINT");
            }
            if (string.IsNullOrWhiteSpace(azureKey))
            {
                throw new ArgumentNullException(nameof(azureKey), "Missing system environment variable: AZURE_COGNITIVE_SERVICE_KEY");
            }

            _client = new TextAnalyticsClient(new Uri(azureEndpoint), new AzureKeyCredential(azureKey));
        }

        public async Task<Response<PiiEntityCollection>> RecognizePiiEntitiesAsync(string? document, string? language)
        {
            var piiEntities = await _client.RecognizePiiEntitiesAsync(document, language);
            return piiEntities;
        }

    }
}


The UI codebehind of the razor component page showing the UI looks like this:

Home.razor.cs


using Azure;
using Microsoft.AspNetCore.Components;
using PiiDetectionDemo.Models;
using PiiDetectionDemo.Util;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace PiiDetectionDemo.Components.Pages
{
    public partial class Home
    {

        private IndexModel Model = new();
        private bool isProcessing = false;
        private bool isSearchPerformed = false;

        private async Task Submit()
        {
            isSearchPerformed = false;
            isProcessing = true;
            try
            {
                var response = await _piiRemovalTextAnalyticsClientService.RecognizePiiEntitiesAsync(Model.InputText, null);
                Model.RedactedText = response?.Value?.RedactedText;
                Model.UpdateHtmlRedactedText();
                Model.AnalysisResult = response?.Value;
                StateHasChanged();
            }
            catch (Exception ex)
            {
                await Console.Out.WriteLineAsync(ex.ToString());
            }
            isProcessing = false;
            isSearchPerformed = true;
        }

        private void removeWhitespace(ChangeEventArgs args)
        {
            Model.InputText = args.Value?.ToString()?.CleanupAllWhiteSpace();
            StateHasChanged();
        }



    }
}



To get the redacted or censored text void of any Pii that the Pii detection feature was able to detect, access the Value of type Azure.AI.TextAnalytics.PiiEntityCollection. Inside this object, the string RedactedText contains the censored / redacted text. The IndexModel looks like this :


using Azure.AI.TextAnalytics;
using Microsoft.AspNetCore.Components;
using PiiDetectionDemo.Util;
using System.ComponentModel.DataAnnotations;
using System.Text;

namespace PiiDetectionDemo.Models
{

    public class IndexModel
    {

        [Required]
        public string? InputText { get; set; }

        public string? RedactedText { get; set; }

        public string? HtmlRedactedText { get; set; }

        public MarkupString HtmlRedactedTextMarkupString { get; set; }

        public void UpdateHtmlRedactedText()
        {
            var sb = new StringBuilder(RedactedText);
            if (AnalysisResult != null && RedactedText != null)
            {
                foreach (var piiEntity in AnalysisResult.OrderByDescending(a => a.Offset))
                {
                    sb.Insert(piiEntity.Offset + piiEntity.Length, "</b></span>");
                    sb.Insert(piiEntity.Offset, $"<span style='background-color:lightgray;border:1px solid black;corner-radius:2px; color:{GetBackgroundColor(piiEntity)}' title='{piiEntity.Category}: {piiEntity.SubCategory} Confidence: {piiEntity.ConfidenceScore} Redacted Text: {piiEntity.Text}'><b>");
                }
            }
            HtmlRedactedText = sb.ToString()?.CleanupAllWhiteSpace();    
            HtmlRedactedTextMarkupString = new MarkupString(HtmlRedactedText ?? string.Empty);
        }

        private string GetBackgroundColor(PiiEntity piiEntity)
        {
            if (piiEntity.Category == PiiEntityCategory.PhoneNumber)
            {
                return "yellow";
            }
            if (piiEntity.Category == PiiEntityCategory.Organization)
            {
                return "orange";
            }
            if (piiEntity.Category == PiiEntityCategory.Address)
            {
                return "green";
            }
            return "gray";                   
        }

        public long ExecutionTime { get; set; }
        public PiiEntityCollection? AnalysisResult { get; set; }

    }
}




Frontend UI looks like this: Home.razor


@page "/"
@using PiiDetectionDemo.Util

@inject IPiiRemovalTextAnalyticsClientService _piiRemovalTextAnalyticsClientService;

<h3>Azure HealthCare Text Analysis - Pii detection feature - Azure Cognitive Services</h3>

<em>Pii = Person identifiable information</em>

<EditForm Model="@Model" OnValidSubmit="@Submit">
    <DataAnnotationsValidator />
    <ValidationSummary />

    <div class="form-group row">
        <label><strong>Text input</strong></label>
        <InputTextArea @oninput="removeWhitespace" class="overflow-scroll" style="max-height:500px;max-width:900px;font-size: 10pt;font-family:Verdana, Geneva, Tahoma, sans-serif" @bind-Value="@Model.InputText" rows="5" />
    </div>

    <div class="form-group row">
        <div class="col">
            <br />
            <button class="btn btn-outline-primary" type="submit">Run</button>
        </div>
        <div class="col">
        </div>
        <div class="col">
        </div>
    </div>

    <br />

    @if (isProcessing)
    {

        <div class="progress" style="max-width: 90%">
            <div class="progress-bar progress-bar-striped progress-bar-animated"
                 style="width: 100%; background-color: green">
                Retrieving result from Azure Text Analysis Pii detection feature. Processing..
            </div>
        </div>
        <br />

    }

    <div class="form-group row">
        <label><strong>Analysis result</strong></label>

        @if (isSearchPerformed)
        {
            <br />
            <b>Execution time took: @Model.ExecutionTime ms (milliseconds)</b>

            <br />
            <br />

            <b>Redacted text (Pii removed)</b>
            <br />

            <div class="form-group row">
               <label><strong>Categorized Pii redacted text</strong></label>
               <div>
               @Model.HtmlRedactedTextMarkupString
               </div>
            </div>

            <br />
            <br />

            <table class="table table-striped table-dark table-hover">
                <thead>
                <th>Pii text</th>
                <th>Category</th>
                <th>SubCategory</th>
                <th>Offset</th>
                <th>Length</th>
                <th>ConfidenceScore</th>
                </thead>
                <tbody>
                    @if (Model.AnalysisResult != null) {
                        @foreach (var entity in Model.AnalysisResult)
                        {
                            <tr>
                                <td>@entity.Text</td>
                                <td>@entity.Category.ToString()</td>
                                <td>@entity.SubCategory</td>
                                <td>@entity.Offset</td>
                                <td>@entity.Length</td>
                                <td>@entity.ConfidenceScore</td>                                        
                            </tr>
                        }
                    }
                </tbody>
            </table>

        }
    </div>

</EditForm>



The Demo uses Bootstrap 5 to build up a HTML table styled and showing the Azure.AI.TextAnalytics.PiiEntity properties.

Sunday, 14 April 2024

Building a filter via Expression trees in C#

This article will look at how to build a filter with Expression trees in C#.

It is an academic exercise how to use Expression trees, you would probably use filters just specifying lambda function with LINQ, but the code shows how you can build an Expression incrementally and compile it to a function. If there is a use-case where LINQ does not fit, perhaps some late-binding scenario or where LINQ does not offer an operator, you can use the approach shows in this article, but the article shows simple usage of Expression trees for introducing Expression trees to C# developers wanting to
test them out in more detail.

The sample code below shows the sample code testing out how to build the Expression incrementally using extension method loading some sample data. An important gotcha is to keep sending in same the objectParameter which is the parameter expression used in the lambda function that is built up, this must be the same parameter. Consider some lambda function of an object 'Person' where the parameter 'x' like:

x => x.Age > 3 && x.Age < 9

The point is that the ParameterExpression x must be the same object, or else we get an error.

FilterHelper.cs


public static class FilterHelper {

	public enum ComparisonOperator {
		Equal,
		LessThan,
		LessThanOrEqual,
		GreaterThan,
		GreaterThanOrEqual,
		NotEqual		
	}
	
	public static Func<TClass, bool> CompileFilter<TClass>(this Expression expression, ParameterExpression objectParameter){
		var expr = Expression.Lambda<Func<TClass, bool>>(expression, false, new List<ParameterExpression>{ objectParameter });
		return expr.Compile();
	}
	
	public static Expression BuildFilter<TClass, TProp>(this Expression previousExpression, Expression<Func<TClass, TProp>> prop, object value,
		ComparisonOperator op, ParameterExpression objectParameter){
		
		var propertyInfo = GetPropertyInfo(prop);
		var propertyToCall = Expression.Property(objectParameter, propertyInfo);		
		var valueToTest = Expression.Constant(value);
		
		Expression operatorExpression = null;
		switch (op)
		{
			case ComparisonOperator.Equal:
				operatorExpression = Expression.Equal(propertyToCall, valueToTest);
				break;
			case ComparisonOperator.NotEqual:
				operatorExpression = Expression.NotEqual(propertyToCall, valueToTest);
				break;
			case ComparisonOperator.LessThan:
				operatorExpression = Expression.LessThan(propertyToCall, valueToTest);
				break;
			case ComparisonOperator.LessThanOrEqual:
				operatorExpression = Expression.LessThanOrEqual(propertyToCall, valueToTest);
				break;
			case ComparisonOperator.GreaterThan:
				operatorExpression = Expression.GreaterThan(propertyToCall, valueToTest);
				break;
			case ComparisonOperator.GreaterThanOrEqual:
				operatorExpression = Expression.GreaterThanOrEqual(propertyToCall, valueToTest);
				break;
		}
		
		if (previousExpression == null){
			return operatorExpression;
		}
		else {
			return Expression.AndAlso(previousExpression, operatorExpression);
		}			
	}

	/// <summary>
	/// Gets the corresponding <see cref="PropertyInfo" /> from an <see cref="Expression" />.
	/// </summary>
	/// <param name="property">The expression that selects the property to get info on.</param>
	/// <returns>The property info collected from the expression.</returns>
	/// <exception cref="ArgumentNullException">When <paramref name="property" /> is <c>null</c>.</exception>
	/// <exception cref="ArgumentException">The expression doesn't indicate a valid property."</exception>
	private static PropertyInfo GetPropertyInfo<T, P>(Expression<Func<T, P>> property)
	{
		if (property == null)
		{
			throw new ArgumentNullException(nameof(property));
		}

		if (property.Body is UnaryExpression unaryExp)
		{
			if (unaryExp.Operand is MemberExpression memberExp)
			{
				return (PropertyInfo)memberExp.Member;
			}
		}
		else if (property.Body is MemberExpression memberExp)
		{
			return (PropertyInfo)memberExp.Member;
		}

		throw new ArgumentException($"The expression doesn't indicate a valid property. [ {property} ]");
	}

}




The sample data uses a POCO Employee as entity class: Employee.cs


public class Employee
{
	public int Id { get; set; }
	public string FirstName { get; set; }
	public string LastName { get; set; }
	public string Department { get; set; }
	public string Position { get; set; }
	public decimal Salary { get; set; }
	public DateTime HireDate { get; set; }
	public DateOnly HireDateOnly
	{
		get { return DateOnly.FromDateTime(HireDate); }
	}
}





The sample program loads up the Json data, then it builds the expression with method BuildFilter shown above and then finally calls CompileFilter to build the expression into a Func<TClass, bool> where TClass is the employee type.

Program.cs


void Main()
{
	string json = File.ReadAllText(Path.Combine(@"C:\Users\SomeUser\Documents\LINQPad Queries\SampleData\Employees.json"));
	var employees = JsonSerializer.Deserialize<List<Employee>>(json, 
new JsonSerializerOptions { PropertyNameCaseInsensitive = true }); //employees.Dump(); var objectParameter = Expression.Parameter(typeof(Employee)); Expression currentFilter = null; currentFilter = FilterHelper.BuildFilter<Employee, object>(currentFilter, e => e.Department,
"Engineering", FilterHelper.ComparisonOperator.Equal, objectParameter) .BuildFilter<Employee, object>(e => e.Salary, 79000m,
FilterHelper.ComparisonOperator.GreaterThan, objectParameter); Func<Employee, bool> employeeFilter = currentFilter.CompileFilter<Employee>(objectParameter); var matchingEmployees = employees.Where(employeeFilter).ToList(); matchingEmployees.Dump(); }


Sample data json looks like this - an array of employees

Employees.json


[
    {
      "id": 1,
      "firstName": "Alice",
      "lastName": "Johnson",
      "department": "HR",
      "position": "Manager",
      "salary": 60000,
      "hireDate": "2022-03-15"
    },
    {
      "id": 2,
      "firstName": "Bob",
      "lastName": "Smith",
      "department": "Engineering",
      "position": "Software Engineer",
      "salary": 80000,
      "hireDate": "2021-09-10"
    },
    {
      "id": 3,
      "firstName": "Charlie",
      "lastName": "Brown",
      "department": "Finance",
      "position": "Financial Analyst",
      "salary": 70000,
      "hireDate": "2020-05-20"
    },
    {
      "id": 4,
      "firstName": "David",
      "lastName": "Lee",
      "department": "Marketing",
      "position": "Marketing Specialist",
      "salary": 65000,
      "hireDate": "2019-11-05"
    },
    {
      "id": 5,
      "firstName": "Eva",
      "lastName": "Garcia",
      "department": "Sales",
      "position": "Sales Representative",
      "salary": 75000,
      "hireDate": "2018-07-12"
    },
    {
      "id": 6,
      "firstName": "Frank",
      "lastName": "Wang",
      "department": "Engineering",
      "position": "Senior Developer",
      "salary": 95000,
      "hireDate": "2017-02-28"
    },
    {
      "id": 7,
      "firstName": "Grace",
      "lastName": "Miller",
      "department": "HR",
      "position": "Recruiter",
      "salary": 55000,
      "hireDate": "2016-08-18"
    },
    {
      "id": 8,
      "firstName": "Henry",
      "lastName": "Chen",
      "department": "Finance",
      "position": "Financial Manager",
      "salary": 90000,
      "hireDate": "2015-04-03"
    },
    {
      "id": 9,
      "firstName": "Ivy",
      "lastName": "Nguyen",
      "department": "Marketing",
      "position": "Content Writer",
      "salary": 60000,
      "hireDate": "2014-10-22"
    },
    {
      "id": 10,
      "firstName": "Jack",
      "lastName": "Kim",
      "department": "Sales",
      "position": "Account Executive",
      "salary": 80000,
      "hireDate": "2013-06-14"
    },
    {
      "id": 11,
      "firstName": "Karen",
      "lastName": "Taylor",
      "department": "Engineering",
      "position": "QA Engineer",
      "salary": 75000,
      "hireDate": "2012-01-09"
    },
    {
      "id": 12,
      "firstName": "Leo",
      "lastName": "Rodriguez",
      "department": "HR",
      "position": "HR Specialist",
      "salary": 55000,
      "hireDate": "2011-07-27"
    },
    {
      "id": 13,
      "firstName": "Mia",
      "lastName": "Liu",
      "department": "Finance",
      "position": "Financial Advisor",
      "salary": 70000,
      "hireDate": "2010-03-16"
    },
    {
      "id": 14,
      "firstName": "Nina",
      "lastName": "Martinez",
      "department": "Marketing",
      "position": "Social Media Manager",
      "salary": 65000,
      "hireDate": "2009-09-05"
    },
    {
      "id": 15,
      "firstName": "Oscar",
      "lastName": "Hernandez",
      "department": "Sales",
      "position": "Sales Manager",
      "salary": 100000,
      "hireDate": "2008-04-21"
    }
  ]



The filter is more limited than just sticking to LINQ, but the code in this example shows how you can build a filter incrementally. Traditionally, you would use Linq and an IEnumerable of TClass and you can keep on filter it too. Here are some closing arguments for why you could make use of Expression trees and have to use them too and not be able to use Linq:
Purpose: Expression trees represent code as data structures. They allow you to build executable code dynamically in C#. Use Cases: - Dynamic Code Generation: When you need to create or modify code at runtime (e.g., building custom queries or transformations). - Remote Execution: Expression trees are useful for scenarios where you want to send calculations across the wire (e.g., database queries, web services). - Custom Query Providers: If you’re building your own query provider (like LINQ to SQL or Entity Framework), expression trees help translate queries into other formats (e.g., SQL).