Showing posts with label .NET 8. Show all posts
Showing posts with label .NET 8. Show all posts

Monday, 5 May 2025

Using MatPlotLib from .NET

MatPlotLib is a powerful library for data visualization. It provides graphing for scientific computing. It can be used for doing both mathematical calculations and statistics. Together with additional libraries like NumPy or Numerical Python, it is clear that Python as a programming language and ecosystem provides a lot of powerful functionality that is also free to use. MatplotLib has a BSD license, which means it can be ued for personal, academic or commercial purposes without restrictions. This article will look at using MatplotLib from .NET. First off an image that displays the demo and example of using MatplotLib.

The source code shown in this article is available on Github here:

https://github.com/toreaurstadboss/SeabornBlazorVisualizer



Using MatPlotLib from .NET

First off, install Anaconda. Anaconda is a Python distribution that contains a large collection of data visualization libraries. A compatible version with the lastest version of Python.net Nuget library. The demo displayed here uses Anaconda version 2023.03.

Anaconda archived versions 2023.03 can be installed from here. Windows users can download the file: https://repo.anaconda.com/archive/Anaconda3-2023.03-1-Windows-x86_64.exe

https://repo.anaconda.com/archive/

Next up, install also Python 3.10 version. It will be used together with Anaconda. A 64-bit installer can be found here:

Python 3.10 installer (Windows 64-bits) The correct versions of NumPy and MatPlotLib can be checked against this list :

https://github.com/toreaurstadboss/SeabornBlazorVisualizer/blob/main/SeabornBlazorVisualizer/conda_list_loading_matplotlib_working_1st_May_2025.txt

Calculating the determinite integral of a function

The demo in this article show in the link at the top has got an appsettings.json file, you can adjust to your environment.

appsettings.json Application configuration file




{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "PythonConfig": {
    "PythonDllPath": "C:\\Python310\\Python310.dll",
    "PythonHome": "C:\\Programdata\\anaconda3",
    "PythonSitePackages":  "C:\\Programdata\\anaconda3\\lib\\site-packages",
    "PythonVersion": "3.10"
  },
  "AllowedHosts": "*"
}


Clone the source code and run the application. It is a Blazor server app. You can run it from VS 2022 for example. The following code shows how Python.net is set up to start using Python. Both Python 3.10 and Anaconda site libs are used here. The Python runtime and engine is set up using this helper class.

PythonInitializer.cs



using Microsoft.Extensions.Options;
using Python.Runtime;

namespace SeabornBlazorVisualizer.Data
{

    /// <summary>
    /// Helper class to initialize the Python runtime
    /// </summary>
    public static class PythonInitializer
    {

        private static bool runtime_initialized = false;

        /// <summary>
        /// Perform one-time initialization of Python runtime
        /// </summary>
        /// <param name="pythonConfig"></param>
        public static void InitializePythonRuntime(IOptions<PythonConfig> pythonConfig)
        {
            if (runtime_initialized)
                return;
            var config = pythonConfig.Value;

            // Set environment variables
            Environment.SetEnvironmentVariable("PYTHONHOME", config.PythonHome, EnvironmentVariableTarget.Process);
            Environment.SetEnvironmentVariable("PYTHONPATH", config.PythonSitePackages, EnvironmentVariableTarget.Process);
            Environment.SetEnvironmentVariable("PYTHONNET_PYDLL", config.PythonDllPath);
            Environment.SetEnvironmentVariable("PYTHONNET_PYVER", config.PythonVersion);

            PythonEngine.Initialize();

            PythonEngine.PythonHome = config.PythonHome ?? Environment.GetEnvironmentVariable("PYTHONHOME", EnvironmentVariableTarget.Process)!;
            PythonEngine.PythonPath = config.PythonDllPath ?? Environment.GetEnvironmentVariable("PYTHONNET_PYDLL", EnvironmentVariableTarget.Process)!;

            PythonEngine.BeginAllowThreads();
            AddSitePackagesToPythonPath(pythonConfig);
            runtime_initialized = true;
        }

        private static void AddSitePackagesToPythonPath(IOptions<PythonConfig> pythonConfig)
        {
            if (!runtime_initialized)
            {
                using (Py.GIL())
                {
                    dynamic sys = Py.Import("sys");
                    sys.path.append(pythonConfig.Value.PythonSitePackages);
                    Console.WriteLine(sys.path);

                    //add folders in solution this too with scripts
                    sys.path.append(@"Data/");
                }
            }
        }

    }
}



The following helper class sets up the site libraries we will use.

PythonHelper.cs



using Python.Runtime;

namespace SeabornBlazorVisualizer.Data
{

    /// <summary>
    /// Helper class to initialize the Python runtime
    /// </summary>
    public static class PythonHelper
    {

        /// <summary>
        /// Imports Python modules. Returned are the following modules:
        /// <para>np (numpy)</para>
        /// <para>os (OS module - standard library)</para>
        /// <para>scipy (scipy)</para>
        /// <para>mpl (matplotlib)</para>
        /// <para>plt (matplotlib.pyplot </para>
        /// </summary>
        /// <returns>Tuple of Python modules</returns>
        public static (dynamic np, dynamic os, dynamic scipy, dynamic mpl, dynamic plt) ImportPythonModules()
        {

            dynamic np = Py.Import("numpy");
            dynamic os = Py.Import("os");
            dynamic mpl = Py.Import("matplotlib");
            dynamic plt = Py.Import("matplotlib.pyplot");
            dynamic scipy = Py.Import("scipy");

            mpl.use("Agg");

            return (np, os, scipy, mpl, plt);
        }

    }
}



The demo is a Blazor server app. The following service will generate the plot of a determinite integral using MatPlotLib. The service saves the plot into a PNG file. This PNG file is saved into the folder wwwroot. The Blazor server app displays the image that was generated and saved.

MatPlotImageService.cs



using Microsoft.Extensions.Options;
using Python.Runtime;

namespace SeabornBlazorVisualizer.Data
{
    public class MatplotPlotImageService
    {

        private IOptions<PythonConfig>? _pythonConfig;

        private static readonly object _lock = new object();

        public MatplotPlotImageService(IOptions<PythonConfig> pythonConfig)
        {
            _pythonConfig = pythonConfig;
            PythonInitializer.InitializePythonRuntime(_pythonConfig);
        }

        public Task<string> GenerateDefiniteIntegral(string functionExpression, int lowerBound, int upperBound)
        {

            string? result = null;

            using (Py.GIL()) // Ensure thread safety for Python calls
            {
                dynamic np = Py.Import("numpy");
                dynamic plt = Py.Import("matplotlib.pyplot");

                dynamic patches = Py.Import("matplotlib.patches"); // Import patches module

                // Create a Python execution scope
                using (var scope = Py.CreateScope())
                {
                    // Define the function inside the scope
                    scope.Exec($@"
import numpy as np
def func(x):
    return {functionExpression}
");

                    // Retrieve function reference from scope
                    dynamic func = scope.Get("func");

                    // Define integration limits
                    double a = lowerBound, b = upperBound;

                    // Generate x-values
                    dynamic x = np.linspace(0, 10, 100); //generate evenly spaced values in range [0, 20], 100 values (per 0.1)
                    dynamic y = func.Invoke(x);

                    // Create plot figure
                    var fig = plt.figure();
                    var ax = fig.add_subplot(111);

                    // set title to function expression
                    plt.title(functionExpression);

                    ax.plot(x, y, "r", linewidth: 2);
                    ax.set_ylim(0, null);

                    // Select range for integral shading
                    dynamic ix = np.linspace(a, b, 100);
                    dynamic iy = func.Invoke(ix);

                    // **Fix: Separate x and y coordinates properly**
                    List<double> xCoords = new List<double> { a }; // Start at (a, 0)
                    List<double> yCoords = new List<double> { 0 };

                    int length = (int)np.size(ix);
                    for (int i = 0; i < length; i++)
                    {
                        xCoords.Add((double)ix[i]);
                        yCoords.Add((double)iy[i]);
                    }

                    xCoords.Add(b); // End at (b, 0)
                    yCoords.Add(0);

                    // Convert x and y lists to NumPy arrays
                    dynamic npVerts = np.column_stack(new object[] { np.array(xCoords), np.array(yCoords) });

                    // **Correctly Instantiate Polygon Using NumPy Array**
                    dynamic poly = patches.Polygon(npVerts, facecolor: "0.6", edgecolor: "0.2");
                    ax.add_patch(poly);

                    // Compute integral area
                    double area = np.trapezoid(iy, ix);
                    ax.text(0.5 * (a + b), 30, "$\\int_a^b f(x)\\mathrm{d}x$", ha: "center", fontsize: 20);
                    ax.text(0.5 * (a + b), 10, $"Area = {area:F2}", ha: "center", fontsize: 12);

                    plt.show();


                    result = SavePlot(plt, dpi: 150);
                }
            }
            return Task.FromResult(result);
        }

        public Task<string> GenerateHistogram(List<double> values, string title = "Provide Plot title", string xlabel = "Provide xlabel title", string ylabel = "Provide ylabel title")
        {
            string? result = null;
            using (Py.GIL()) //Python Global Interpreter Lock (GIL)
            {
                var (np, os, scipy, mpl, plt) = PythonHelper.ImportPythonModules();

                var distribution = np.array(values.ToArray());

                //// Ensure clearing the plot
                //plt.clf();

                var fig = plt.figure(); //create a new figure
                var ax1 = fig.add_subplot(1, 2, 1);
                var ax2 = fig.add_subplot(1, 2, 2);

                // Add style
                plt.style.use("ggplot");

                var counts_bins_patches = ax1.hist(distribution, edgecolor: "black");

                // Normalize counts to get colors 
                var counts = counts_bins_patches[0];
                var patches = counts_bins_patches[2];

                var norm_counts = counts / np.max(counts);

                int norm_counts_size = Convert.ToInt32(norm_counts.size.ToString());

                // Apply colors to patches based on frequency
                for (int i = 0; i < norm_counts_size; i++)
                {
                    plt.setp(patches[i], "facecolor", plt.cm.viridis(norm_counts[i])); //plt.cm is the colormap module in MatPlotlib. viridis creates color maps from normalized value 0 to 1 that is optimized for color-blind people.
                }

                // **** AX1 Histogram first - frequency counts ***** 

                ax1.set_title(title);
                ax1.set_xlabel(xlabel);
                ax1.set_ylabel(ylabel);

                string cwd = os.getcwd();

                // Calculate average and standard deviation
                var average = np.mean(distribution);
                var std_dev = np.std(distribution);
                var total_count = np.size(distribution);

                // Format average and standard deviation to two decimal places
                var average_formatted = np.round(average, 2);
                var std_dev_formatted = np.round(std_dev, 2);

                //Add legend with average and standard deviation
                ax1.legend(new string[] { $"Total count: {total_count}\n Average: {average_formatted} cm\nStd Dev: {std_dev_formatted} cm" }, framealpha: 0.5, fancybox: true);



                //***** AX2 : Set up ax2 = Percentage histogram next *******

                ax2.set_title("Percentage distribution");
                ax2.set_xlabel(xlabel);
                ax2.set_ylabel(ylabel);
                // Fix for CS1977: Cast the lambda expression to a delegate type
                ax2.yaxis.set_major_formatter((PyObject)plt.FuncFormatter(new Func<double, int, string>((y, _) => $"{y:P0}")));

                ax2.hist(distribution, edgecolor: "black", weights: np.ones(distribution.size) / distribution.size);

                // Format y-axis to show percentages
                ax2.yaxis.set_major_formatter(plt.FuncFormatter(new Func<double, int, string>((y, _) => $"{y:P0}")));

                // tight layout to prevent overlap 
                plt.tight_layout();

                // Show the plot with the two subplots at last (render to back buffer 'Agg', see method SavePlot for details)
                plt.show();

                result = SavePlot(plt, theme: "bmh", dpi: 150);
            }

            return Task.FromResult(result);
        }

        public Task<string> GeneratedCumulativeGraphFromValues(List<double> values)
        {
            string? result = null;
            using (Py.GIL()) //Python Global Interpreter Lock (GIL)
            {
                var (np, os, scipy, mpl, plt) = PythonHelper.ImportPythonModules();

                dynamic pythonValues = np.cumsum(np.array(values.ToArray()));

                // Ensure clearing the plot
                plt.clf();

                // Create a figure with increased size
                dynamic fig = plt.figure(figsize: new PyTuple(new PyObject[] { new PyFloat(6), new PyFloat(4) }));

                // Plot data
                plt.plot(values, color: "green");

                string cwd = os.getcwd();

                result = SavePlot(plt, theme: "ggplot", dpi: 200);

            }

            return Task.FromResult(result);
        }

        public Task<string> GenerateRandomizedCumulativeGraph()
        {
            string? result = null;
            using (Py.GIL()) //Python Global Interpreter Lock (GIL)
            {

                dynamic np = Py.Import("numpy");

                //TODO : Remove imports of pandas and scipy and datetime if they are not needed

                Py.Import("pandas");
                Py.Import("scipy");
                Py.Import("datetime");
                dynamic os = Py.Import("os");

                dynamic mpl = Py.Import("matplotlib");
                dynamic plt = Py.Import("matplotlib.pyplot");

                // Set dark theme
                plt.style.use("ggplot");

                mpl.use("Agg");


                // Generate data
                //dynamic x = np.arange(0, 10, 0.1);
                //dynamic y = np.multiply(2, x); // Use NumPy's multiply function

                dynamic values = np.cumsum(np.random.randn(1000, 1));


                // Ensure clearing the plot
                plt.clf();

                // Create a figure with increased size
                dynamic fig = plt.figure(figsize: new PyTuple(new PyObject[] { new PyFloat(6), new PyFloat(4) }));

                // Plot data
                plt.plot(values, color: "blue");

                string cwd = os.getcwd();

                result = SavePlot(plt, theme: "ggplot", dpi: 200);

            }

            return Task.FromResult(result);
        }

        /// <summary>
        /// Saves the plot to a PNG file with a unique name based on the current date and time
        /// </summary>
        /// <param name="plot">Plot, must be a PyPlot plot use Python.net Py.Import("matplotlib.pyplot")</param>
        /// <param name="theme"></param>
        /// <param name="dpi"></param>
        /// <returns></returns>
        public string? SavePlot(dynamic plt, string theme = "ggplot", int dpi = 200)
        {
            string? plotSavedImagePath = null;
            //using (Py.GIL()) //Python Global Interpreter Lock (GIL)
            //{
            dynamic os = Py.Import("os");
            dynamic mpl = Py.Import("matplotlib");
            // Set dark theme
            plt.style.use(theme);
            mpl.use("Agg"); //set up rendering of plot to back-buffer ('headless' mode)

            string cwd = os.getcwd();
            // Save plot to PNG file
            string imageToCreatePath = $@"GeneratedImages\{DateTime.Now.ToString("yyyyMMddHHmmss")}{Guid.NewGuid().ToString("N")}_plotimg.png";
            string imageToCreateWithFolderPath = $@"{cwd}\wwwroot\{imageToCreatePath}";
            plt.savefig(imageToCreateWithFolderPath, dpi: dpi); //save the plot to a file (use full path)
            plotSavedImagePath = imageToCreatePath;

            CleanupOldGeneratedImages(cwd);
            //}
            return plotSavedImagePath;
        }

        private static void CleanupOldGeneratedImages(string cwd)
        {
            lock (_lock)
            {

                Directory.GetFiles(cwd + @"\wwwroot\GeneratedImages", "*.png")
                 .OrderByDescending(File.GetLastWriteTime)
                 .Skip(10)
                 .ToList()
                 .ForEach(File.Delete);
            }
        }

}



The code above shows some additional examples of using MatPlotLib.
  • Histogram example
  • Line graph using cumulative sum by making use of NumPy or a helper method in .NET
These examples demonstrates also that MatPlotLib can be used for statistics, which today for .NET is mostly crunched with the help of Excel or EP Plus library for example. Since Python is considered as the home of data visualization with its vast ecosystem of data science libraries, this article and demos shows how you can get started with using this ecosystem from .NET. Note, using Python.net to create these plots in MatPlotLib is best prepared using Jupyter Notebook. When the plot displayed looks okay, it is time to integrate that Python script into .NET and C# using Python.Net library. Make note that there will be some challenges to get the Python code to work in C# of course. When passing in values to a function, sometimes you must use
for example NumPy to create compatible data types. Also note the usage of the Pystatic class here from Python.net , which offers the GIL Global Interpreter lock and a way to import Python modules.

https://jupyter.org/ A screenshot showing histogram in the demo is shown below. As we can see, MatPlotLib can be used from many different data visualizations and domains.

Tuesday, 22 April 2025

Predicting variable using Regression with ML.net

This article will look at regression with ML.net In the example, the variable "Poverty rate" measured as a percentage against amount "teenage pregnancies" per 1,000 birth. The data is fetched from a publicly available CSV file. The data is obtained from Jeff Prosise repos of ML.net here on Github:

https://github.com/jeffprosise/ML.NET/blob/master/MLN-SimpleRegression/MLN-SimpleRegression/Data/poverty.csv



In this article, Linqpad 8 will be used. First off, the following two Nuget packages are added :
  • Microsoft.ML
  • Microsoft.ML.Mkl.Components
The following method will plot a scatter graph from provided MLContext data, and add a standard linear trendline, which will work with the example in this article.

Plotutils.cs



void PlotScatterGraph<T>(MLContext mlContext, IDataView trainData, Func<T, PointItem> pointCreator, string chartTitle) where T : class, new()
{
	//Convert the IDataview to an enumerable collection
	var data = mlContext.Data.CreateEnumerable<T>(trainData, reuseRowObject: false).Select(x => pointCreator(x)).ToList();

	// Calculate trendline (simple linear regression)
	double avgX = data.Average(d => d.X);
	double avgY = data.Average(d => d.Y);
	double slope = data.Sum(d => (d.X - avgX) * (d.Y - avgY)) / data.Sum(d => (d.X - avgX) * (d.X - avgX));
	double intercept = avgY - slope * avgX;
	var trendline = data.Select(d => new { X = d.X, Y = slope * d.X + intercept }).ToList();

	//Plot the scatter graph
	var plot = data.Chart(d => d.X)
		.AddYSeries(d => d.Y, LINQPad.Util.SeriesType.Point, chartTitle)
		.AddYSeries(d => trendline.FirstOrDefault(t => t.X == d.X)?.Y ?? 0, Util.SeriesType.Line, "Trendline")
		.ToWindowsChart();
		
	plot.AntiAliasing = System.Windows.Forms.DataVisualization.Charting.AntiAliasingStyles.All;
	plot.Dump();
}



Let's look at the code for loading the CSV data and into the MLContext and then used the method TrainTestSplit to split the data into training data and testing data. Note also the classes Input and Output and the usage of LoadColumn and ColumnName

Program.cs



void Main()
{

	string inputFile = Path.Combine(Path.GetDirectoryName(Util.CurrentQueryPath)!, @"Sampledata\poverty2.csv"); //linqpad tech

	var context = new MLContext(seed: 0);

	//Train the model 
	var data = context.Data
		.LoadFromTextFile<Input>(inputFile, hasHeader: true, separatorChar: ';');
	
	// Split data into training and test sets 
	var split = context.Data.TrainTestSplit(data, testFraction: 0.2);
	var trainData = split.TrainSet;
	var testData = split.TestSet;

	var pipeline = context
		.Transforms.NormalizeMinMax("PovertyRate")
		.Append(context.Transforms.Concatenate("Features", "PovertyRate"))
		.Append(context.Regression.Trainers.Ols());

	var model = pipeline.Fit(trainData);
	// Use the model to make a prediction
	var predictor = context.Model.CreatePredictionEngine<Input, Output>(model);
	var input = new Input { PovertyRate = 8.4f };

	var actual = 36.8f;

	var prediction = predictor.Predict(input);
	Console.WriteLine($"Input poverty rate: {input.PovertyRate} . Predicted birth rate per 1000: {prediction.TeenageBirthRate:0.##}");
	Console.WriteLine($"Actual birth rate per 1000: {actual}");

	// Evaluate the regression model 
	var predictions = model.Transform(testData);
	var metrics = context.Regression.Evaluate(predictions);
	Console.WriteLine($"R-squared: {metrics.RSquared:0.##}");
	Console.WriteLine($"Root Mean Squared Error: {metrics.RootMeanSquaredError:0.##}");
	Console.WriteLine($"Mean Absolute Error: {metrics.MeanAbsoluteError:0.##}");
	Console.WriteLine($"Mean Squared Error: {metrics.MeanSquaredError:0.##}");


	PlotScatterGraph<Input>(context, trainData, (Input input) => 
		new PointItem { X = (float) Math.Round(input.PovertyRate, 2), Y = (float) Math.Round(input.TeenageBirthRate, 2) },
		"Poverty rate (%) vs Teenage Pregnancies per 1,000 birth");

}

public class PointItem {
	public float X { get; set; }
	public float Y { get; set; }
}

void PlotScatterGraph<T>(MLContext mlContext, IDataView trainData, Func<T, PointItem> pointCreator, string chartTitle) where T : class, new()
{
	//Convert the IDataview to an enumerable collection
	var data = mlContext.Data.CreateEnumerable<T>(trainData, reuseRowObject: false).Select(x => pointCreator(x)).ToList();

	// Calculate trendline (simple linear regression)
	double avgX = data.Average(d => d.X);
	double avgY = data.Average(d => d.Y);
	double slope = data.Sum(d => (d.X - avgX) * (d.Y - avgY)) / data.Sum(d => (d.X - avgX) * (d.X - avgX));
	double intercept = avgY - slope * avgX;
	var trendline = data.Select(d => new { X = d.X, Y = slope * d.X + intercept }).ToList();

	//Plot the scatter graph
	var plot = data.Chart(d => d.X)
		.AddYSeries(d => d.Y, LINQPad.Util.SeriesType.Point, chartTitle)
		.AddYSeries(d => trendline.FirstOrDefault(t => t.X == d.X)?.Y ?? 0, Util.SeriesType.Line, "Trendline")
		.ToWindowsChart();
		
	plot.AntiAliasing = System.Windows.Forms.DataVisualization.Charting.AntiAliasingStyles.All;
	plot.Dump();
}



public class Input
{

	[LoadColumn(1)]
	public float PovertyRate;

	[LoadColumn(5), ColumnName("Label")]
	public float TeenageBirthRate { get; set; }

}
public class Output
{
	[ColumnName("Score")]
	public float TeenageBirthRate;

}


A pipeline is defined for the machine learning here consisting of the following :
  • The method NormalizeMinMax will transform the poverty rate into a normalized scale between 0 and 1. The Concatenate method will be used to specify the "Features", in this case only the column Poverty rate is the feature of which we want to predict a score, this is the rate of teenage pregnancy births per 1,000 births. Note that our CSV data set contains more columns, but this is a simple regression where only one variable is taken into account.
  • The trainers used to train the machine learning algorithm is Ols, the Ordinary Least Squares.
  • The method fit will train using the training data defined from the method TrainTestSplit.
  • The resulting model is used to create a prediction engine.
  • Using the prediction engine, it is possible to predict a value value using the Predict method given one input item. Our prediction engine expects input objects of type Input and Output.
  • Using the testdata, the method Transform using the model gives us multiple predictions and it is possible to evalute the regression analysis from the predictions to check how accurate the regression model is.
  • Returning from this evaluation, we get the R-squared for example. This is a value from 0 to 1.0 where it describes how accurate the regression is in in describing the total variation of the residues of the model, the amount the data when plotted in a scatter graph where residue is the offset between the actual data and what the regression model predicts.
  • Other values such as RMSE and MSE are the root and mean squared error, which are absolute values.
  • Using the code above we got a fairly accurate regression model, but more accuracy would be achieved by taking in additional factors.


  • Output from the Linqpad 8 application shown in this article :
    
        
    Input poverty rate: 8,4 . Predicted birth rate per 1000: 35,06
    Actual birth rate per 1000: 36,8
    R-squared: 0,59
    Root Mean Squared Error: 8,99
    Mean Absolute Error: 8,01
    Mean Squared Error: 80,83
        
      
    
    Please note that there are some standard column names used for machine learning.
    
    Label: Represents the target variable (the value to predict).
    
    Features: Contains the input features used for training.
    
    Score: Stores the predicted value in regression models.
    
    PredictedLabel: Holds the predicted class in classification models.
    
    Probability: Represents the probability of a predicted class.
    
    FeatureContributions: Shows how much each feature contributes to a prediction.
    
    
    In the code above, the column names "Label", "Features" and "Score" was used to instruct the regression being calculated in the code here for ML.Net context model. The attribute ColumnName was being used here together with the Concatenate method.

Tuesday, 15 April 2025

Adding plugins to use for Semantic Kernel

With Microsoft Semantic Kernel, it is possible to consume AI services such as Azure AI and OpenAI with less code, as this framework provides simplification and standardization for consuming these services. A repo on Github with the code shown in this article is provided here :

https://github.com/toreaurstadboss/SemanticKernelPluginDemov4

The demo code is a Blazor server app. It demonstrates how to use Microsoft Semantic Kernel with plugins. I have decided to provide the Northwind database as the extra data the plugin will use. Via debugging and seeing the output, I see that the plugin is successfully called and used. It is also easy to add plugins, which provides additional data to the AI model. This is suitable for providing AI powered solutions with private data that you want to provide to the AI model. For example, when using OpenAI Chat GPT-4, providing a plugin will make it possible to specify which data are to be presented and displayed. It is a convenient way to provide a natural language interface for doing data reporting such as listing up results from this plugin. The plugin can provide kernel functions, using attributes on the method. Let's first look at the Program.cs file for wiring up the semantic kernel for a Blazor Server demo app.

Program.cs



using Microsoft.EntityFrameworkCore;
using Microsoft.SemanticKernel;
using SemanticKernelPluginDemov4.Models;
using SemanticKernelPluginDemov4.Services;


namespace SemanticKernelPluginDemov4
{
    public class Program
    {
        public static void Main(string[] args)
        {
            var builder = WebApplication.CreateBuilder(args);

            // Add services to the container.
            builder.Services.AddRazorPages();
            builder.Services.AddServerSideBlazor();

            // Add DbContext
            builder.Services.AddDbContextFactory<NorthwindContext>(options =>
                options.UseSqlServer(builder.Configuration.GetConnectionString("DefaultConnection")));

            builder.Services.AddScoped<IOpenAIChatcompletionService, OpenAIChatcompletionService>();

            builder.Services.AddScoped<NorthwindSemanticKernelPlugin>();

            builder.Services.AddScoped(sp =>
            {
                var kernelBuilder = Kernel.CreateBuilder();
                kernelBuilder.AddOpenAIChatCompletion(modelId: builder.Configuration.GetSection("OpenAI").GetValue<string>("ModelId")!,
                    apiKey: builder.Configuration.GetSection("OpenAI").GetValue<string>("ApiKey")!);

                var kernel = kernelBuilder.Build();

                var dbContextFactory = sp.GetRequiredService<IDbContextFactory<NorthwindContext>>();
                var northwindSemanticKernelPlugin = new NorthwindSemanticKernelPlugin(dbContextFactory);
                kernel.ImportPluginFromObject(northwindSemanticKernelPlugin);

                return kernel;
            });

            var app = builder.Build();

            // Configure the HTTP request pipeline.
            if (!app.Environment.IsDevelopment())
            {
                app.UseExceptionHandler("/Error");
                // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
                app.UseHsts();
            }

            app.UseHttpsRedirection();

            app.UseStaticFiles();

            app.UseRouting();

            app.MapBlazorHub();
            app.MapFallbackToPage("/_Host");

            app.Run();
        }
    }
}



In the code above, note the following:
  • The usage of IDbContextFactory for creating a db context, injected into the plugin. This is a Blazor server app, so this service is used to create db contet, since a Blazor server will have a durable connection between client and the server over Signal-R and there needs to use this interface to create dbcontext instances as needed
  • Using the method ImportPluginFromObject to import the plugin into the semantic kernel built here. Note that we register the kernel as a scoped service here. Also the plugin is registered as a scoped service here.
The plugin looks like this.

NorthwindSemanticKernelplugin.cs



using Microsoft.EntityFrameworkCore;
using Microsoft.SemanticKernel;
using SemanticKernelPluginDemov4.Models;
using System.ComponentModel;

namespace SemanticKernelPluginDemov4.Services
{

    public class NorthwindSemanticKernelPlugin
    {
        private readonly IDbContextFactory<NorthwindContext> _northwindContext;

        public NorthwindSemanticKernelPlugin(IDbContextFactory<NorthwindContext> northwindContext)
        {
            _northwindContext = northwindContext;
        }

        [KernelFunction]
        [Description("When asked about the suppliers of Nortwind database, use this method to get all the suppliers. Inform that the data comes from the Semantic Kernel plugin called : NortwindSemanticKernelPlugin")]
        public async Task<List<string>> GetSuppliers()
        {
            using (var dbContext = _northwindContext.CreateDbContext())
            {
                return await dbContext.Suppliers.OrderBy(s => s.CompanyName).Select(s => "Kernel method 'NorthwindSemanticKernelPlugin:GetSuppliers' gave this: " + s.CompanyName).ToListAsync();
            }
        }

        [KernelFunction]
        [Description("When asked about the total sales of a given month in a year, use this method. In case asked for multiple months, call this method again multiple times, adjusting the month and year as provided. The month and year is to be in the range 1-12 for months and for year 1996-1998. Suggest for the user what the valid ranges are in case other values are provided.")]
        public async Task<decimal> GetTotalSalesInMontAndYear(int month, int year)
        {
            using (var dbContext = _northwindContext.CreateDbContext())
            {
                var sumOfOrders = await (from Order o in dbContext.Orders
                             join OrderDetail od in dbContext.OrderDetails on o.OrderId equals od.OrderId
                             where o.OrderDate.HasValue && (o.OrderDate.Value.Month == month
                             && o.OrderDate.Value.Year == year) 
                             select (od.UnitPrice * od.Quantity) * (1 - (decimal)od.Discount)).SumAsync();

                return sumOfOrders;
            }
        }

    }
}


In the code above, note the attributes used. KernelFunction tells that this is a method the Semantic kernel can use. The description attribute instructs the AI LLM model how to use the method, how to provide parameter values if any and when the method is to be called. Let's look at the OpenAI service next.

OpenAIChatcompletionService.cs



using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.ChatCompletion;
using Microsoft.SemanticKernel.Connectors.OpenAI;

namespace SemanticKernelPluginDemov4.Services
{

    public class OpenAIChatcompletionService : IOpenAIChatcompletionService
    {
        private readonly Kernel _kernel;

        private IChatCompletionService _chatCompletionService;

        public OpenAIChatcompletionService(Kernel kernel)
        {
            _kernel = kernel;
            _chatCompletionService = kernel.GetRequiredService<IChatCompletionService>();
        }

        public async IAsyncEnumerable<string?> RunQuery(string question)
        {
            var chatHistory = new ChatHistory();

            chatHistory.AddSystemMessage("You are a helpful assistant, answering only on questions about Northwind database. In case you got other questions, inform that you only can provide questions about the Northwind database. It is important that only the provided Northwind database functions added to the language model through plugin is used when answering the questions. If no answer is available, inform this.");

            chatHistory.AddUserMessage(question);

            await foreach (var chatUpdate in _chatCompletionService.GetStreamingChatMessageContentsAsync(chatHistory, CreateOpenAIExecutionSettings(), _kernel))
            {
                yield return chatUpdate.Content;
            }
        }

        private OpenAIPromptExecutionSettings? CreateOpenAIExecutionSettings()
        {
            return new OpenAIPromptExecutionSettings
            {
                ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions
            };
        }

    }
}



In the code above, the kernel is injected into this service. The kernel was registered in Program.cs as a scoped service, so it is injected here. The method GetRequiredService is similar to the method with same name of IServiceProvider used inside Program.cs. Note the use of ToolCallBehavior set to AutoInvokeKernelFunctions. Extending the AI powered functionality with plugins requires little extra code with Microsoft Semantic kernel. A screenshot of the demo is shown below.

Sunday, 9 March 2025

Generating dropdowns for enums in Blazor

This article will look into generating dropdown for enums in Blazor. The repository for the source code listed in the article is here: https://github.com/toreaurstadboss/DallEImageGenerationImgeDemoV4 First off, a helper class for enums that will use the InputSelect control. The helper class will support setting the display text for enum options / alternatives via resources files using the display attribute.

Enumhelper.cs | C# source code



using DallEImageGenerationImageDemoV4.Models;
using Microsoft.AspNetCore.Components;
using Microsoft.AspNetCore.Components.Forms;
using System.ComponentModel.DataAnnotations;
using System.Linq.Expressions;
using System.Resources;

namespace DallEImageGenerationImageDemoV4.Utility
{
  
    public static class EnumHelper
    {
      
        public static RenderFragment GenerateEnumDropDown<TEnum>(
            object receiver,
            TEnum selectedValue,
            Action<TEnum> valueChanged) 
            where TEnum : Enum
        {
            Expression<Func<TEnum>> onValueExpression = () => selectedValue;
            var onValueChanged = EventCallback.Factory.Create<TEnum>(receiver, valueChanged);
            return builder =>
            {
                // Set the selectedValue to the first enum value if it is not set
                if (EqualityComparer<TEnum>.Default.Equals(selectedValue, default))
                {
                    object? firstEnum = Enum.GetValues(typeof(TEnum)).GetValue(0);
                    if (firstEnum != null)
                    {
                        selectedValue = (TEnum)firstEnum;
                    }
                }

                builder.OpenComponent<InputSelect<TEnum>>(0);
                builder.AddAttribute(1, "Value", selectedValue);
                builder.AddAttribute(2, "ValueChanged", onValueChanged);
                builder.AddAttribute(3, "ValueExpression", onValueExpression);
                builder.AddAttribute(4, "class", "form-select");  // Adding Bootstrap class for styling
                builder.AddAttribute(5, "ChildContent", (RenderFragment)(childBuilder =>
                {
                    foreach (var value in Enum.GetValues(typeof(TEnum)))
                    {
                        childBuilder.OpenElement(6, "option");
                        childBuilder.AddAttribute(7, "value", value?.ToString());
                        childBuilder.AddContent(8, GetEnumOptionDisplayText(value)?.ToString()?.Replace("_", " ")); // Ensure the display text is clean
                        childBuilder.CloseElement();
                    }
                }));
                builder.CloseComponent();
            };
        }

        /// <summary>
        /// Retrieves the display text of an enum alternative 
        /// </summary>
        private static string? GetEnumOptionDisplayText<T>(T value)
        {
            string? result = value!.ToString()!; 

            var displayAttribute = value
                .GetType()
                .GetField(value!.ToString()!)
                ?.GetCustomAttributes(typeof(DisplayAttribute), false)?
                .OfType<DisplayAttribute>()
                .FirstOrDefault();
            if (displayAttribute != null)
            {
                if (displayAttribute.ResourceType != null && !string.IsNullOrWhiteSpace(displayAttribute.Name))
                {
                    result = new ResourceManager(displayAttribute.ResourceType).GetString(displayAttribute!.Name!);                    
                }
                else if (!string.IsNullOrWhiteSpace(displayAttribute.Name))
                {
                    result = displayAttribute.Name;
                }           
            }
            return result;          
        }


    }
}



The following razor component shows how to use this helper.


 <div class="form-group">
     <label for="Quality" class="form-class fw-bold">GeneratedImageQuality</label>
     @EnumHelper.GenerateEnumDropDown(this, homeModel.Quality,v => homeModel.Quality = v)
     <ValidationMessage For="@(() => homeModel.Quality)" class="text-danger" />
 </div>
 <div class="form-group">
     <label for="Size" class="form-label fw-bold">GeneratedImageSize</label>
     @EnumHelper.GenerateEnumDropDown(this, homeModel.Size, v => homeModel.Size = v)
     <ValidationMessage For="@(() => homeModel.Size)" class="text-danger" />
 </div>
 <div class="form-group">
     <label for="Style" class="form-label fw-bold">GeneratedImageStyle</label>
     @EnumHelper.GenerateEnumDropDown(this, homeModel.Style, v => homeModel.Style = v)
     <ValidationMessage For="@(() => homeModel.Style)" class="text-danger" />
 </div>


It would be possible to instead make a component than such a helper method that just passes a typeref parameter of the enum type. But using such a programmatic helper returning a RenderFragment. As the code shows, returning a builder which uses the RenderTreeBuilder let's you register the rendertree to return here. It is possible to use OpenComponent and CloseComponent. Using AddAttribute to add attributes to the InputSelect. And a childbuilder for the option values. Sometimes it is easier to just make such a class with helper method instead of a component. The downside is that it is a more manual process, it is similar to how MVC uses HtmlHelpers. What is the best option from using a component or such a RenderFragment helper is not clear, it is a technique many developers using Blazor should be aware of.

Monday, 16 December 2024

SpeechService in Azure AI Text to Speech

This article will present Azure AI Text To Speech service. The code for this article is available to clone from Github repo here:

https://github.com/toreaurstadboss/SpeechSynthesis.git

The speech service uses AI trained speech to provide natural speech and ease of use. You can just provide text and get it read out aloud. An overview of supported languages in the Speech service is shown here:

https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=stt

Azure AI Speech Synthesis DEMO

You can create a TTS - Text To Speech service using Azure AI service for this. This Speech service in this demo uses the library Nuget Microsoft.CognitiveServices.Speech.

This repo contains a simple demo using Azure AI Speech synthesis using Azure.CognitiveServices.SpeechSynthesis.
It provides a simple way of synthesizing text to speech using Azure AI services. Its usage is shown here:

The code provides a simple builder for creating a SpeechSynthesizer instance.

using Microsoft.CognitiveServices.Speech;

namespace ToreAurstadIT.AzureAIDemo.SpeechSynthesis;

public class Program
{
    private static async Task Main(string[] args)
    {
        Console.WriteLine("Your text to speech input");
        string? text = Console.ReadLine();

        using (var synthesizer = SpeechSynthesizerBuilder.Instance.WithSubscription().Build())
        {
            using (var result = await synthesizer.SpeakTextAsync(text))
            {
                string reasonResult = result.Reason switch
                {
                    ResultReason.SynthesizingAudioCompleted => $"The following text succeeded successfully: {text}",
                    _ => $"Result of speeech synthesis: {result.Reason}"
                };
                Console.WriteLine(reasonResult);
            }
        }

    }

}

The builder looks like this:
using Microsoft.CognitiveServices.Speech;

namespace ToreAurstadIT.AzureAIDemo.SpeechSynthesis;

public class SpeechSynthesizerBuilder
{

    private string? _subscriptionKey = null;
    private string? _subscriptionRegion = null;

    public static SpeechSynthesizerBuilder Instance => new SpeechSynthesizerBuilder();

    public SpeechSynthesizerBuilder WithSubscription(string? subscriptionKey = null, string? region = null)
    {
        _subscriptionKey = subscriptionKey ?? Environment.GetEnvironmentVariable("AZURE_AI_SERVICES_SPEECH_KEY", EnvironmentVariableTarget.User);
        _subscriptionRegion = region ?? Environment.GetEnvironmentVariable("AZURE_AI_SERVICES_SPEECH_REGION", EnvironmentVariableTarget.User);
        return this;
    }

    public SpeechSynthesizer Build()
    {
        var config = SpeechConfig.FromSubscription(_subscriptionKey, _subscriptionRegion);
        var speechSynthesizer = new SpeechSynthesizer(config);
        return speechSynthesizer;
    }
}

Note that I observed that the audio could get chopped off in the very end. It might be a temporary issue, but if you encounter it too, you can add an initial pause to avoid this:

   string? intialPause = "     ....     "; //this is added to avoid the text being cut in the start

Sunday, 8 December 2024

Extending Azure AI Search with data sources

This article will present both code and tips around getting Azure AI Search to utilize additional data sources. The article builds upon the previous article in the blog:

https://toreaurstad.blogspot.com/2024/12/azure-ai-openai-chat-gpt-4-client.html

This code will use Open AI Chat GPT-4 together with additional data source. I have tested this using Storage account in Azure which contains blobs with documents. First off, create Azure AI services if you do not have this yet.



Then create an Azure AI Search



Choose the location and the Pricing Tier. You can choose the Free (F) pricing tier to test out the Azure AI Search. The standard pricing tier comes in at about 250 USD per month, so a word of caution here as billing might incur if you do not choose the Free tier. Head over to the Azure AI Search service after it is crated and note inside the Overview the Url. Expand the Search management and choose the folowing menu options and fill out them in this order:
  • Data sources
  • Indexes
  • Indexers


There are several types of data sources you can add.
  • Azure Blog Storage
  • Azure Data Lake Storage Gen2
  • Azure Cosmos DB
  • Azure SQL Database
  • Azure Table Storage
  • Fabric OneLake files

Upload files to the blob container

  • I have tested out adding a data source using Azure Blob Storage. I had to create a new storage account and I believe Azure might have changed it over the years, so for best compability, add a brand new storage account. Then choose a blob container inside the Blob storage, then hit the Create button.
  • Head over to your Storage browser inside your storage account, then choose Blob container. You can add a Blob container and then after it is created, click the Upload button.
  • You can then upload multiple files into the blob container (it is like a folder, which saves your files as blobs).

Setting up the index

  • After the Blob storage (storage account) is added to the data source, choose the Indexes menu button inside Azure AI search. Click Add index.
  • After the index is added, choose the button Add field
  • Add a field name called : Edit.String of type Edm.String.
  • Click the checkbox for Retrievable and Searchable. Click the button Save

Setting up the indexer

  • Choose to add an Indexer via button Add indexer
  • Choose the Index you added
  • Choose the Data source you added
  • Select the indexed extensions and specify which file types to index. Probably you should select text based files here, such as .md and .markdown files and even some binary file type such as .pdf and .docx can be selected here
  • Data to extract: Choose Content and metadata


Source code for this article

The source code can be cloned from this Github repo:
br /> https://github.com/toreaurstadboss/OpenAIDemo.git

The code for this article is available in the branch:
feature/openai-search-documentsources To add the data source to our ChatClient instance, we do the following. Please note that this method will be changed in the Azure AI SDK in the future :


            ChatCompletionOptions? chatCompletionOptions = null;
            if (dataSources?.Any() == true)
            {
                chatCompletionOptions = new ChatCompletionOptions();

                foreach (var dataSource in dataSources!)
                {
#pragma warning disable AOAI001 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
                    chatCompletionOptions.AddDataSource(new AzureSearchChatDataSource()
                    {
                        Endpoint = new Uri(dataSource.endpoint),
                        IndexName = dataSource.indexname,
                        Authentication = DataSourceAuthentication.FromApiKey(dataSource.authentication)
                    });
#pragma warning restore AOAI001 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
                }

            }
            




The updated version of the extension class of OpenAI.Chat.ChatClient then looks like this: ChatClientExtensions.cs



using Azure.AI.OpenAI.Chat;
using OpenAI.Chat;
using System.ClientModel;
using System.Text;

namespace ToreAurstadIT.OpenAIDemo
{
    public static class ChatclientExtensions
    {

        /// <summary>
        /// Provides a stream result from the Chatclient service using AzureAI services.
        /// </summary>
        /// <param name="chatClient">ChatClient instance</param>
        /// <param name="message">The message to send and communicate to the ai-model</param>
        /// <returns>Streamed chat reply / result. Consume using 'await foreach'</returns>
        public static AsyncCollectionResult<StreamingChatCompletionUpdate> GetStreamedReplyAsync(this ChatClient chatClient, string message,
            (string endpoint, string indexname, string authentication)[]? dataSources = null)
        {
            ChatCompletionOptions? chatCompletionOptions = null;
            if (dataSources?.Any() == true)
            {
                chatCompletionOptions = new ChatCompletionOptions();

                foreach (var dataSource in dataSources!)
                {
#pragma warning disable AOAI001 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
                    chatCompletionOptions.AddDataSource(new AzureSearchChatDataSource()
                    {
                        Endpoint = new Uri(dataSource.endpoint),
                        IndexName = dataSource.indexname,
                        Authentication = DataSourceAuthentication.FromApiKey(dataSource.authentication)
                    });
#pragma warning restore AOAI001 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
                }

            }

            return chatClient.CompleteChatStreamingAsync(
                [new SystemChatMessage("You are an helpful, wonderful AI assistant"), new UserChatMessage(message)], chatCompletionOptions);
        }

        public static async Task<string> GetStreamedReplyStringAsync(this ChatClient chatClient, string message, (string endpoint, string indexname, string authentication)[]? dataSources = null, bool outputToConsole = false)
        {
            var sb = new StringBuilder();
            await foreach (var update in GetStreamedReplyAsync(chatClient, message, dataSources))
            {
                foreach (var textReply in update.ContentUpdate.Select(cu => cu.Text))
                {
                    sb.Append(textReply);
                    if (outputToConsole)
                    {
                        Console.Write(textReply);
                    }
                }
            }
            return sb.ToString();
        }

    }
}





The updated code for the demo app then looks like this, I chose to just use tuples here for the endpoint, index name and api key:

ChatpGptDemo.cs


using OpenAI.Chat;
using OpenAIDemo;
using System.Diagnostics;

namespace ToreAurstadIT.OpenAIDemo
{
    public class ChatGptDemo
    {

        public async Task<string?> RunChatGptQuery(ChatClient? chatClient, string msg)
        {
            if (chatClient == null)
            {
                Console.WriteLine("Sorry, the demo failed. The chatClient did not initialize propertly.");
                return null;
            }

            Console.WriteLine("Searching ... Please wait..");

            var stopWatch = Stopwatch.StartNew();

            var chatDataSources = new[]{
                (
                    SearchEndPoint: Environment.GetEnvironmentVariable("AZURE_SEARCH_AI_ENDPOINT", EnvironmentVariableTarget.User) ?? "N/A",
                    SearchIndexName: Environment.GetEnvironmentVariable("AZURE_SEARCH_AI_INDEXNAME", EnvironmentVariableTarget.User) ?? "N/A",
                    SearchApiKey: Environment.GetEnvironmentVariable("AZURE_SEARCH_AI_APIKEY", EnvironmentVariableTarget.User) ?? "N/A"
                )
            };

            string reply = "";

            try
            {

                reply = await chatClient.GetStreamedReplyStringAsync(msg, dataSources: chatDataSources, outputToConsole: true);
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.Message);
            }

            Console.WriteLine($"The operation took: {stopWatch.ElapsedMilliseconds} ms");


            Console.WriteLine();

            return reply;
        }

    }
}




The code here expects that three user-specific environment variables exists. Please note that the API key can be found under the menu item Keys in Azure AI Search. There are two admin keys and multiple query keys. To distribute keys to other users, you of course share the API query key, not the admin key(s). The screenshot below shows the demo. It is a console application, it could be web application or other client : Please note that the Free tier of Azure AI Search is rather slow and seems to only allow queryes at a certain interval, it will suffice to just test it out. To really test it out in for example an Intranet scenario, the standard tier Azure AI search service is recommended, at about 250 USD per month as noted.

Conclusions

Getting an Azure AI Chat service to work in intranet scenarios using a combination of Open AI Chat GPT-4 together with a custom collection of files that are indexed offers a nice combination of building up a knowledge base which you can query against. It is rather convenient way of building an on-premise solution for intranet AI chat service using Azure cloud services.

Sunday, 8 September 2024

Using lazy loading in Entity Framework Core 8

This article will show some code how you can opt in something called lazy loading in EF. This means you do not load in all the related data for an entity until you need the data. Lets look at a simple entity called Customer. We will add to navigational properties, that is related entities. Without eager loading enabled automatically or lazy loading enabled automatically, EF Core 8 will not populated these navigational properties, which is pointing to the related entities. The fields will be null without active measure on the loading part. Let's inspect how to lazy load such navigational properties.

Customer.cs




 public class Customer {
 
  // more code.. 
 
  public Customer()
  {
      AddressCustomers = new HashSet<AddressCustomer>();
  }
  
  // more code .. 
 
  private Customer(ILazyLoader lazyLoader)
  {
    LazyLoader = lazyLoader;
  }

  public CustomerRank CustomerRank { get; set; }

  public virtual ICollection<AddressCustomer> AddressCustomers { get; set; }
  
 }
  
  
 
 
 
First off, the ILazyLoader service is from Microsoft.EntityFrameworkCore.Infrastructure. It is injected inside the entity, preferably using a private constructor of the entity. Now you can set up lazy loading a for a navigational property like this :



 public CustomerRank CustomerRank
 {
     get => LazyLoader.Load(this, ref _customerRank);
     set => _customerRank = value;
 }
  
  
 
 
 
If it feels a bit unclean to mix entity code with behavioral code since we inject a service into our domain models or entities, you can use the Fluent api instead while setting up the DbContext.



  modelBuilder.Entity<Customer>()
      .Navigation(e => e.AddressCustomers)
     .AutoInclude();

  modelBuilder.Entity<Customer>(entity =>
  {
      entity.HasKey(e => e.Id);
     entity.Navigation(e => e.CustomerRank).AutoInclude();
  });


 
 
 
If automatically lazy loading the data (the data will be loaded upon access of the navigational property) seems a bit little flexible, one can also set up loading manually wherever in the application code using the methods Entry and either Reference or Collection and then the Load method.



var customer = _dbContext.Customers.First();

_dbContext
    .Entry(customer)
    .Reference(c => c.CustomerRank)
    .Load();


_dbContext
    .Entry(customer)
    .Collection(c => c.AddressCustomers)
    .Load();


Once more, note that the data is still lazy loaded, their content will only be loaded when you access the particular navigational property pointing to the related data. Also note that if you debug in say VS 2022, data might look like they are automatically loaded, but this is because the debugger loads the contents if it can and will even do so for lazy loaded navigational fields. If you instead make in your application code a programmatic access to this navigational property and output the data you will see the data also being loaded, but this happens once it is programatic access. For example if we made the private field _customerRank public (as we should not do to protect our domain model's data) you can see this while debugging :


//changed a field in Customer.cs to become public for external access :
//  public CustomerRank _customerRank;

Console.WriteLine(customer._customerRank);
Console.WriteLine(customer.CustomerRank);

// considering this set up 

  public CustomerRank CustomerRank
  {
      get => LazyLoader.Load(this, ref _customerRank);
      set => _customerRank = value;
  }



The field _customerRank is initially null, it is when we access the property CustomerRank which I set to be AutoInclude i.e. lazy loaded I see that data is loaded.

Thursday, 9 May 2024

Azure Cognitive Synthesized Text To Speech with voice styles

Using Azure Cognitive Services, it is possible to translate text into other languages and also synthesize the text to speech. It is also possible to add voice effects such as style of the voice. This adds more realism by adding emotions to a synthesized voice. The voice is already trained by neural net training and adding voice style makes the synthesized speech even more realistic and multi-purpose. The Github repo for this is available here as .NET Maui Blazor client written with .NET 8 :

MultiLingual translator DEMO Github repo

Not all the voices supported in Azure Cognitive Services do support voice effects. An overview of which voices are shown here:

https://learn.microsoft.com/nb-no/azure/ai-services/speech-service/language-support?tabs=tts#voice-styles-and-roles

More and more synthetic voices in Azure Cognitive Services gets more and more voice styles which express emotions. For now, most of the voices are either english (en-US) or chinese (zh-CN) and a few other languages got some few voices supporting styles. This will most likely be improved into the future where these neural net trained voices are trained in voice styles or some generic voice style algorithm is achieved that can infer emotions on a generic level, although that still sounds a bit sci-fi.

Azure Cognitive Text-To-Speech Voices with support for emotions / voice styles


Voice Styles Roles
de-DE-ConradNeural1 cheerful Not supported
en-GB-SoniaNeural cheerful, sad Not supported
en-US-AriaNeural angry, chat, cheerful, customerservice, empathetic, excited, friendly, hopeful, narration-professional, newscast-casual, newscast-formal, sad, shouting, terrified, unfriendly, whispering Not supported
en-US-DavisNeural angry, chat, cheerful, excited, friendly, hopeful, sad, shouting, terrified, unfriendly, whispering Not supported
en-US-GuyNeural angry, cheerful, excited, friendly, hopeful, newscast, sad, shouting, terrified, unfriendly, whispering Not supported
en-US-JaneNeural angry, cheerful, excited, friendly, hopeful, sad, shouting, terrified, unfriendly, whispering Not supported
en-US-JasonNeural angry, cheerful, excited, friendly, hopeful, sad, shouting, terrified, unfriendly, whispering Not supported
en-US-JennyNeural angry, assistant, chat, cheerful, customerservice, excited, friendly, hopeful, newscast, sad, shouting, terrified, unfriendly, whispering Not supported
en-US-NancyNeural angry, cheerful, excited, friendly, hopeful, sad, shouting, terrified, unfriendly, whispering Not supported
en-US-SaraNeural angry, cheerful, excited, friendly, hopeful, sad, shouting, terrified, unfriendly, whispering Not supported
en-US-TonyNeural angry, cheerful, excited, friendly, hopeful, sad, shouting, terrified, unfriendly, whispering Not supported
es-MX-JorgeNeural chat, cheerful Not supported
fr-FR-DeniseNeural cheerful, sad Not supported
fr-FR-HenriNeural cheerful, sad Not supported
it-IT-IsabellaNeural chat, cheerful Not supported
ja-JP-NanamiNeural chat, cheerful, customerservice Not supported
pt-BR-FranciscaNeural calm Not supported
zh-CN-XiaohanNeural affectionate, angry, calm, cheerful, disgruntled, embarrassed, fearful, gentle, sad, serious Not supported
zh-CN-XiaomengNeural chat Not supported
zh-CN-XiaomoNeural affectionate, angry, calm, cheerful, depressed, disgruntled, embarrassed, envious, fearful, gentle, sad, serious Boy, Girl, OlderAdultFemale, OlderAdultMale, SeniorFemale, SeniorMale, YoungAdultFemale, YoungAdultMale
zh-CN-XiaoruiNeural angry, calm, fearful, sad Not supported
zh-CN-XiaoshuangNeural chat Not supported
zh-CN-XiaoxiaoNeural affectionate, angry, assistant, calm, chat, chat-casual, cheerful, customerservice, disgruntled, fearful, friendly, gentle, lyrical, newscast, poetry-reading, sad, serious, sorry, whisper Not supported
zh-CN-XiaoyiNeural affectionate, angry, cheerful, disgruntled, embarrassed, fearful, gentle, sad, serious Not supported
zh-CN-XiaozhenNeural angry, cheerful, disgruntled, fearful, sad, serious Not supported
zh-CN-YunfengNeural angry, cheerful, depressed, disgruntled, fearful, sad, serious Not supported
zh-CN-YunhaoNeural2 advertisement-upbeat Not supported
zh-CN-YunjianNeural3,4 angry, cheerful, depressed, disgruntled, documentary-narration, narration-relaxed, sad, serious, sports-commentary, sports-commentary-excited Not supported
zh-CN-YunxiaNeural angry, calm, cheerful, fearful, sad Not supported
zh-CN-YunxiNeural angry, assistant, chat, cheerful, depressed, disgruntled, embarrassed, fearful, narration-relaxed, newscast, sad, serious Boy, Narrator, YoungAdultMale
zh-CN-YunyangNeural customerservice, narration-professional, newscast-casual Not supported
zh-CN-YunyeNeural angry, calm, cheerful, disgruntled, embarrassed, fearful, sad, serious Boy, Girl, OlderAdultFemale, OlderAdultMale, SeniorFemale, SeniorMale, YoungAdultFemale, YoungAdultMale
zh-CN-YunzeNeural angry, calm, cheerful, depressed, disgruntled, documentary-narration, fearful, sad, serious OlderAdultMale, SeniorMale

Screenshot from the DEMO showing its user interface. You enter the text to translate at the top and the language of the text is detected using Azure Cognitive Services text detection functionality. And you can then select which language to translate the text into. It will call a REST call to Azure Cognitive Services to translate the text. And it is also possible to hear the speech of the text. Now, it is also added to add voice style. Use the table shown above to select a voice actor that supports a voice style you want to test. As noted, voice styles are still limited to a few languages and voice actors supporting emotions or voice styles. You will hear the voice from the voice actor in a normal mood or voice style if additional emotions or voice styles are not supported.
Let's look at some code for this DEMO too. You can study the Github repo and clone it to test it out yourself. The TextToSpeechUtil class handles much of the logic of creating voice from text input and also create the SSML-XML contents and performt the REST api call to create the voice file. Note that SSML mentioned here, is the Speech Synthesis Markup Language (SSML). The SSML standard is documented here on MSDN, it is a standard adopted by others too including Google.

https://learn.microsoft.com/en-us/azure/ai-services/speech-service/speech-synthesis-markup



using Microsoft.Extensions.Configuration;
using MultiLingual.Translator.Lib.Models;
using System;
using System.Security;
using System.Text;
using System.Xml.Linq;
using static System.Runtime.InteropServices.JavaScript.JSType;

namespace MultiLingual.Translator.Lib
{
    public class TextToSpeechUtil : ITextToSpeechUtil
    {

        public TextToSpeechUtil(IConfiguration configuration)
        {
            _configuration = configuration;
        }

        public async Task<TextToSpeechResult> GetSpeechFromText(string text, string language, TextToSpeechLanguage[] actorVoices, 
            string? preferredVoiceActorId, string? preferredVoiceStyle)
        {
            var result = new TextToSpeechResult();

            result.Transcript = GetSpeechTextXml(text, language, actorVoices, preferredVoiceActorId, preferredVoiceStyle, result);
            result.ContentType = _configuration[TextToSpeechSpeechContentType];
            result.OutputFormat = _configuration[TextToSpeechSpeechXMicrosoftOutputFormat];
            result.UserAgent = _configuration[TextToSpeechSpeechUserAgent];
            result.AvailableVoiceActorIds = ResolveAvailableActorVoiceIds(language, actorVoices);
            result.LanguageCode = language;

            string? token = await GetUpdatedToken();

            HttpClient httpClient = GetTextToSpeechWebClient(token);

            string ttsEndpointUrl = _configuration[TextToSpeechSpeechEndpoint];
            var response = await httpClient.PostAsync(ttsEndpointUrl, new StringContent(result.Transcript, Encoding.UTF8, result.ContentType));

            using (var memStream = new MemoryStream()) {
                var responseStream = await response.Content.ReadAsStreamAsync();
                responseStream.CopyTo(memStream);
                result.VoiceData = memStream.ToArray();
            }

            return result;
        }

        private async Task<string?> GetUpdatedToken()
        {
            string? token = _token?.ToNormalString();
            if (_lastTimeTokenFetched == null || DateTime.Now.Subtract(_lastTimeTokenFetched.Value).Minutes > 8)
            {
                token = await GetIssuedToken();
            }

            return token;
        }

        private HttpClient GetTextToSpeechWebClient(string? token)
        {
            var httpClient = new HttpClient();
            httpClient.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", token);
            httpClient.DefaultRequestHeaders.Add("X-Microsoft-OutputFormat", _configuration[TextToSpeechSpeechXMicrosoftOutputFormat]);
            httpClient.DefaultRequestHeaders.Add("User-Agent", _configuration[TextToSpeechSpeechUserAgent]);
            return httpClient;
        }
       
        public string GetSpeechTextXml(string text, string language, TextToSpeechLanguage[] actorVoices, string? preferredVoiceActorId,
              string? preferredVoiceStyle, TextToSpeechResult result)
        {
            result.VoiceActorId = ResolveVoiceActorId(language, preferredVoiceActorId, actorVoices);
            string speechXml = $@"
            <speak version='1.0' xml:lang='en-US' xmlns:mstts='https://www.w3.org/2001/mstts'>
                <voice xml:gender='Male' name='Microsoft Server Speech Text to Speech Voice {result.VoiceActorId}'>
                    <prosody rate='1'>{text}</prosody>
                </voice>
            </speak>";

            speechXml = AddVoiceStyleEffectIfDesired(preferredVoiceStyle, speechXml);

            return speechXml;
        }

        /// <summary>
        /// Adds voice style / expression to the SSML markup for the voice
        /// </summary>
        private static string AddVoiceStyleEffectIfDesired(string? preferredVoiceStyle, string speechXml)
        {
            if (!string.IsNullOrWhiteSpace(preferredVoiceStyle) && preferredVoiceStyle != "normal-neutral")
            {
                var voiceDoc = XDocument.Parse(speechXml); //https://learn.microsoft.com/nb-no/azure/ai-services/speech-service/speech-synthesis-markup-voice#use-speaking-styles-and-roles

                XElement? prosody = voiceDoc.Descendants("prosody").FirstOrDefault();
                if (prosody?.Value != null)
                {
                    // Create the <mstts:express-as> element, for now skip the ':' letter and replace at the end

                    var expressedAsWrappedElement = new XElement("msttsexpress-as",
                        new XAttribute("style", preferredVoiceStyle));
                    expressedAsWrappedElement.Value = prosody!.Value;
                    prosody?.ReplaceWith(expressedAsWrappedElement);
                    speechXml = voiceDoc.ToString().Replace(@"msttsexpress-as", "mstts:express-as");
                }
            }

            return speechXml;
        }

        private List<string> ResolveAvailableActorVoiceIds(string language, TextToSpeechLanguage[] actorVoices)
        {
            if (actorVoices?.Any() == true)
            {
                var voiceActorIds = actorVoices.Where(v => v.LanguageKey == language || v.LanguageKey.Split("-")[0] == language).SelectMany(v => v.VoiceActors).Select(v => v.VoiceId).ToList();
                return voiceActorIds;
            }
            return new List<string>();
        }

        private string ResolveVoiceActorId(string language, string? preferredVoiceActorId, TextToSpeechLanguage[] actorVoices)
        {
            string actorVoiceId = "(en-AU, NatashaNeural)"; //default to a select voice actor id 
            if (actorVoices?.Any() == true)
            {
                var voiceActorsForLanguage = actorVoices.Where(v => v.LanguageKey == language || v.LanguageKey.Split("-")[0] == language).SelectMany(v => v.VoiceActors).Select(v => v.VoiceId).ToList();
                if (voiceActorsForLanguage != null)
                {
                    if (voiceActorsForLanguage.Any() == true)
                    {
                        var resolvedPreferredVoiceActorId = voiceActorsForLanguage.FirstOrDefault(v => v == preferredVoiceActorId);
                        if (!string.IsNullOrWhiteSpace(resolvedPreferredVoiceActorId))
                        {
                            return resolvedPreferredVoiceActorId!;
                        }
                        actorVoiceId = voiceActorsForLanguage.First();
                    }
                }
            }
            return actorVoiceId;
        }

        private async Task<string> GetIssuedToken()
        {
            var httpClient = new HttpClient();
            string? textToSpeechSubscriptionKey = Environment.GetEnvironmentVariable("AZURE_TEXT_SPEECH_SUBSCRIPTION_KEY", EnvironmentVariableTarget.Machine);
            httpClient.DefaultRequestHeaders.Add(OcpApiSubscriptionKeyHeaderName, textToSpeechSubscriptionKey);
            string tokenEndpointUrl = _configuration[TextToSpeechIssueTokenEndpoint];
            var response = await httpClient.PostAsync(tokenEndpointUrl, new StringContent("{}"));
            _token = (await response.Content.ReadAsStringAsync()).ToSecureString();
            _lastTimeTokenFetched = DateTime.Now;
            return _token.ToNormalString();
        }

        public async Task<List<string>> GetVoiceStyles()
        {
            var voiceStyles = new List<string>
            {
                "normal-neutral",
                "advertisement_upbeat",
                "affectionate",
                "angry",
                "assistant",
                "calm",
                "chat",
                "cheerful",
                "customerservice",
                "depressed",
                "disgruntled",
                "documentary-narration",
                "embarrassed",
                "empathetic",
                "envious",
                "excited",
                "fearful",
                "friendly",
                "gentle",
                "hopeful",
                "lyrical",
                "narration-professional",
                "narration-relaxed",
                "newscast",
                "newscast-casual",
                "newscast-formal",
                "poetry-reading",
                "sad",
                "serious",
                "shouting",
                "sports_commentary",
                "sports_commentary_excited",
                "whispering",
                "terrified",
                "unfriendly"
            };
            return await Task.FromResult(voiceStyles);
        }

        private const string OcpApiSubscriptionKeyHeaderName = "Ocp-Apim-Subscription-Key";
        private const string TextToSpeechIssueTokenEndpoint = "TextToSpeechIssueTokenEndpoint";
        private const string TextToSpeechSpeechEndpoint = "TextToSpeechSpeechEndpoint";        
        private const string TextToSpeechSpeechContentType = "TextToSpeechSpeechContentType";
        private const string TextToSpeechSpeechUserAgent = "TextToSpeechSpeechUserAgent";
        private const string TextToSpeechSpeechXMicrosoftOutputFormat = "TextToSpeechSpeechXMicrosoftOutputFormat";

        private readonly IConfiguration _configuration;

        private DateTime? _lastTimeTokenFetched = null;
        private SecureString _token = null;

    }
}

 
 

The REST call to generate the voice file is using following set up: TTS endpoint url: https://norwayeast.tts.speech.microsoft.com/cognitiveservices/v1 The transcript (text to translate into speech) is the following in my test as a SSML-XML document:


<speak version="1.0" xml:lang="en-US" xmlns:mstts="https://www.w3.org/2001/mstts">
  <voice xml:gender="Male" name="Microsoft Server Speech Text to Speech Voice (en-US, JaneNeural)">
    <mstts:express-as style="angry">I listen to Eurovision and cheer for Norway</mstts:express-as>
  </voice>
</speak>


The SSML also contains an extension called mstts extension language that adds features to SSML such as the express-as set to a voice style or emotion of "angry". Not all emotions or voice styles are supported by every voice actor in Azure Cognitive Services. But this is a list of the voice styles that could be supported, it varies which voice actor you choose (and inherently which language).
  • "normal-neutral"
  • "advertisement_upbeat"
  • "affectionate"
  • "angry"
  • "assistant"
  • "calm"
  • "chat"
  • "cheerful"
  • "customerservice"
  • "depressed"
  • "disgruntled"
  • "documentary-narration"
  • "embarrassed"
  • "empathetic"
  • "envious"
  • "excited"
  • "fearful"
  • "friendly"
  • "gentle"
  • "hopeful"
  • "lyrical"
  • "narration-professional"
  • "narration-relaxed"
  • "newscast"
  • "newscast-casual"
  • "newscast-formal"
  • "poetry-reading"
  • "sad"
  • "serious"
  • "shouting"
  • "sports_commentary"
  • "sports_commentary_excited"
  • "whispering"
  • "terrified"
  • "unfriendly
Microsoft has come a long way from the early work with SAPI - Microsoft Speech API with Microsoft SAM around 2000. The realism of synthetic voices more than 20 years ago were rather crude and robotic. Nowaydays, voice actors provided by Azure Cloud computing platform as shown here are neural net trained and very realistic based upon training from real voice actors and now more and more voice actor voices support emotions or voice styles. The usages of this can be diverse. Making use of text synthesis can serve in automated answering services and apps in diverse fields such as healthcare and public services or education and more. Making this demo has been fun for me and it can be used to learn languages and with the voice functionality you can train on not only the translation but also pronounciation.