Showing posts with label Blazor. Show all posts
Showing posts with label Blazor. Show all posts

Saturday, 18 January 2025

Monitoring User Secrets inside Blazor

This article shows how you can add User Secrets for a Blazor app, or other related .NET client technology supporting them. User secrets are stored on the individual computer, so one do not have to expose them to others. They can still be shared between different people if they are told what the secrets are, but is practical in many cases where one for example do not want to expose the secrets such as a password, by checking it into
source code repositories. This is due to the fact as mentioned that the user secrets are as noted saved on the individual computer.

User secrets was added in .NET Core 1.0, already released in 2016. Not all developers are familiar with them. Inside Visual Studio 2022, you can right click on the Project of a solution and choose Manage User Secrets. When you choose that option, a file called secrets.json is opened. The file location for this file is shown if you hover over the file. The file location is saved here:

  
    %APPDATA%\Microsoft\UserSecrets\<user_secrets_id>\secrets.json
  
  
You can find the id here, the user_secrets_id, inside the project file (the .csproj file).
Example of such a setting below:
  
  <Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <UserSecretsId>339fab44-57cf-400c-89f9-46e037bb0392</UserSecretsId>
  </PropertyGroup>

</Project>

  

Let's first look at the way we can set up user secrets inside a startup file for the application. Note the usage of reloadOnChange set to true. And adding the user secrets as a singleton service wrapped inside IOptionsMonitor.

Program.cs



builder.Configuration.Sources.Clear();
builder.Configuration
        .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
        .AddJsonFile($"appsettings.{builder.Environment.EnvironmentName}.json", optional: true, reloadOnChange: true)
        .AddUserSecrets(Assembly.GetEntryAssembly()!, optional:false, reloadOnChange: true)
        .AddEnvironmentVariables();

builder.Services.Configure<ModelSecrets>(builder.Configuration.GetSection("ModelSecrets"));
builder.Services.AddSingleton<IOptionsMonitor<ModelSecrets>>, OptionsMonitor<ModelSecrets>>();



The Model secrets class looks like the following.



namespace StoringSecrets
{
    public class ModelSecrets
    {
        public string ApiKey { get; set; } = string.Empty;
    }
}


Home.razor.cs

Let's then look how we can fetch values from the user secrets. The code file for a file Home.razor is put in a file of its own, Home.razor.cs


using Microsoft.AspNetCore.Components;
using Microsoft.Extensions.Options;

namespace ToreAurstadIt.StoringSecrets.Components.Pages
{
    public partial class Home
    {
        [Inject]
        public required IOptionsMonitor<ModelSecrets> ModelSecretsMonitor { get; set; }

        public ModelSecrets? ModelSecrets  { get; set; }

        protected override async Task OnInitializedAsync()
        {
            ModelSecrets = ModelSecretsMonitor.CurrentValue;
            ModelSecretsMonitor.OnChange(updatedSecrets =>
            {
                ModelSecrets = updatedSecrets;
                InvokeAsync(StateHasChanged);

            });
            await Task.CompletedTask;
        }
    }
}

Note that IOptionsMonitor<ModelSecrets> is injected here and that in the OnInitializedAsync method, the injected value uses the OnChange method and and action callback then sets the value of the ModelSecrets and calls InvokeAsync method with StateHasChanged. We output the CurrentValue into the razor view of the Blazor app.

Home.razor



@page "/"

<PageTitle>Home</PageTitle>

<h1>Hello, world!</h1>

Welcome to your new app.

Your user secret is: 
<div>
    @ModelSecrets?.ApiKey
</div>


The secret.json file looks like this:

secrets.json



{
  "ModelSecrets": {
    "ApiKey": "SAMPLE_VALUE_UPDATE_9"
  }
}


A small app shows how this can be done, by changing the user secrets file and then reloading the page, changes should be immediately seen:

Saturday, 4 January 2025

Slider component for Blazor

I have added a Blazor component for Blazor, which uses INPUT of type range and additional CSS styling for more flexible setup of look and feel. The Blazor component is available on Github in my repo here:

https://github.com/toreaurstadboss/BlazorSlider

Blazor lib component

This repository contains Blazor lib Slider component that shows an input of type 'range'.

The slider got default horizontal layout, where the minimum value for the slider is shown to the most left of the scale, which goes along the x-axis for the slider got towards higher values and the maximum value is the value to the most right. The slider x-axis goes along the 'slider track'.

The value of the slider is indicated by the 'slider thumb'. Below the slider are shown 'tick marks', which are controlled by the Minimum and Maximum values and StepSize. Note that the supported data types are the data types that are IConvertible and struct, and the code expects types that can be converted to double. You can use integers for example, but also decimals or floats and so on. In addition, enums can be used, but it works only if your enum got consecutive values, for example 0,1,2,3,4 . The best results are if these consecutive values got the same StepSize. To start using the Blazor slider, add this using in your .razor file where you want to use the component.
 
@using BlazorSliderLib

Please note that the slider has been tested using Bootstrap, more specifically this version:

"bootstrap@5.3.3"
Here is sample markup you can add to test out the Blazor slider (3 sliders are rendered using a custom model and the updated values are shown in labels below :

    <div class="container"> 

        <div class="row">
            <div class="form-control col-md-4">
                <p><b>EQ5D-5L question 1.</b> <br />Mobility. Ability to walk.</p>
                <BlazorSliderLib.Slider T="Eq5dWalk" UseAlternateStyle="AlternateStyle.AlternateStyleInverseColorScale" Title="Ability to walk" ValueChanged="@((e) => UpdateEq5dQ1(e))"
                MinimumDescription="No Problems = The best ability to walk you can imagine" MaximumDescription="Incapable = The worst ability to walk you can imagine" />
            </div>
        </div>

        <div class="row">
            <div class="form-control col-md-4">
                <p><b>EQ5D-5L question 6.</b> <br />We would like to how good or bad your health is TODAY.</p>
            </div>
        </div>

        <div class="row">
            <div class="form-control col-md-4">
                <BlazorSliderLib.Slider T="int" UseAlternateStyle="AlternateStyle.AlternateStyle" Minimum="0" Maximum="100" @bind-Value="@(Model.Data.Eq5dq6)" Stepsize="5" Title="Your health today"
                MinimumDescription="0 = The worst health you can imagine" MaximumDescription="100 = The best health you can imagine" />
            </div>
        </div>

        <div class="row">
            <div class="form-control col-md-4">
                <p><b>EQ5D-5L question 6.</b> <br />We would like to how good or bad your health is TODAY. V2 field.</p>
            </div>
        </div>

        <div class="row">
            <div class="form-control col-md-4">
                <BlazorSliderLib.Slider T="int" Minimum="0" Maximum="100" ValueChanged="@((e) => UpdateEq5dq6V2(e))" Stepsize="5" Title="Your health today (v2 field)"
                MinimumDescription="0 = The worst health you can imagine" MaximumDescription="100 = The best health you can imagine" />
            </div>
        </div>

        <div class="row">
            <div class="form-control col-md-4">
                <p>Value of Model.Data.Eq5dq1</p>
                @Model.Data.Eq5dq1
            </div>
        </div>

        <div class="row">
            <div class="form-control col-md-4"> <p>Value of Model.Data.Eq5d6</p>
                @Model.Data.Eq5dq6 
            </div> 
        </div>

        <div class="row">
            <div class="form-control col-md-4">
                <p>Value of Model.Data.Eq5d6V2</p>
                @Model.Data.Eq5dq6V2
            </div>
        </div>

    </div>

The different setup of sliders

The slider is set up either with an alternate style or using the default styling for sliders, that is, the slider uses an input type of 'range' and the default documented styling on Mozilla Developer Network (MDN) to render a Blazor slider. In addition, it is possible to set up the alternate style to use a inverted color range where higher values will get a reddish color and lower values will get a greenish color. The standard alternate style will show greenish colors for higher values. The following screenshot shows the possible styling that is possible. Note that the default styling is shown in the slider at the bottom, which will render a bit different in different browsers. In Chrome for example, the slider will render with a bluish color. In Edge Chromium, a grayish color is used for the 'slider tick' and 'slider thumb'. Screenshots showing the sliders: The following parameters can be used:
Title
Required. The title is shown below the slider component and centered horizontally along the center of the x-axis which the slider is oriented.
Value
The value of the slider. It can be data bound using either the @bind-Value directive attribute that supports two-way data binding. You can instead use the @ValueChanged event callback, if desired.
Minimum
The minimum value along the slider. It is default set to 0 for numbers. For enums, the lowest value is chosen of the enum (minimum enum alternative, converted to double internally).
Maximum
The maximum value along the slider. It is default set to 100 for numbers. For enums, the higheset value is chosen of the enum (maximum enum alternative, converted to double internally).
Stepsize
The step size for the slider. It is default set to 5 for numbers. For enums, it is set to 1. (note that internally the slider must use double values to work with the _tickmarks_, which expects double values).
ShowTickmarks
Shows tick marks for slider. It is default set to 'true'. Tick marks are generated from the values of Minimum, Maximum and StepSize.
MinimumDescription
Shows additionally description for the minimum value, shown as a small label below the slider. It will only be shown in the value is not empty.
UseAlternateStyle
If the UseAlternateStyle is set to either AlternateStyle and AlternateStyleInverseColorScale, alternate styling is used.

CSS rules to enable the slider

Actually, it is necessary to define a set of CSS rules to make the slider work. The slider's css rules are defined in two different files.

Default CSS rules

`Slider.css` The CSS rules below are taken from MDN Mozilla Developer Network page for the input type 'range' control. Input type 'range' control MDN article:

https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/range

Additional settings are set up. The width is set to 100% so the slider can get as much horizontal space as possible and 'stretch'. There are also basic styles set up for both the tick label and datalist.The datalist is the tickmarks for the slider. The tick marks are automatically generated for the slider.


.sliderv2
{
    width:100%;
}

.sliderv2Label {
    font-weight: 400;
    text-align: center;
    left: 50%;
    font-size:0.7em;
    font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, margin-bottom: 2px;
}

datalist {
    display: flex;
    flex-direction: column;
    justify-content: space-between;
    writing-mode: vertical-lr;
    width: 100%;
}

.tick-label {

    justify-content: space-between;
    font-size:0.6em;

    top: 20px; /* Adjust this value as needed */
}

input[type="range"] {
    width: 100%;
    margin: 0;
}


Alternate CSS rules

`SliderAlternate.css` The alternate CSS rules are setting up additional styling, where color encoding is used for the 'slider track' where higher values along the 'slider track' get a more 'greenish color', while lower values gets 'reddish values'. It is possible to set up the inverse color encoding here, with higher values getting 'reddish color'. Lower values gets more 'greenish colors' in this setup.


.alternate-style input[type="range"] {
    -webkit-appearance: none; /* Remove default styling */
    width: 100%;
    height: 8px;
    background: #ddd;
    outline: none;
    opacity: 0.7;
    transition: opacity .2s;
}

    .alternate-style input[type="range"]:hover {
        opacity: 1;
    }

    .alternate-style input[type="range"]::-webkit-slider-runnable-track {
        width: 100%;
        height: 8px;
        background: linear-gradient(to left, #A5D6A7, #FFF9C4, #FFCDD2); /* More desaturated gradient color */
        border: none;
        border-radius: 3px;
    }

        .alternate-style-inverse-colorscale input[type="range"]::-webkit-slider-runnable-track {
            background: linear-gradient(to right, #A5D6A7, #FFF9C4, #FFCDD2) !important; /* More desaturated gradient color, inverted color range */
        }


.alternate-style input[type="range"]::-webkit-slider-thumb {
    -webkit-appearance: none; /* Remove default styling */
    appearance: none;
    width: 25px;
    height: 25px;
    background: #2E7D32; /* Even darker green thumb color */
    cursor: pointer;
    border-radius: 50%;
    margin-top: -15px !important; /* Move the thumb up */
}

    .alternate-style input[type="range"]::-moz-range-track {
        width: 100%;
        height: 8px;
        background: linear-gradient(to left, #A5D6A7, #FFF9C4, #FFCDD2); /* More desaturated gradient color */
        border: none;
        border-radius: 3px;
    }

        .alternate-style-inverse-colorscale input[type="range"]::-moz-range-track {
            background: linear-gradient(to right, #A5D6A7, #FFF9C4, #FFCDD2 !important; /* More desaturated gradient color, inverted color range */
        }

    .alternate-style input[type="range"]::-moz-range-thumb {
        width: 25px;
        height: 25px;
        background: #2E7D32; /* Even darker green thumb color */
        cursor: pointer;
        border-radius: 50%;
        transform: translateY(-15px); /* Move the thumb up */
    }


The implementation for the Blazor slider looks like this, in the codebehind file for the Slider:


using Microsoft.AspNetCore.Components;

namespace BlazorSliderLib
{

    /// <summary>
    /// Slider to be used in Blazor. Uses input type='range' with HTML5 element datalist and custom css to show a slider.
    /// To add tick marks, set the <see cref="ShowTickmarks"/> to true (this is default)
    /// </summary>
    /// <typeparam name="T"></typeparam>
    public partial class Slider<T> : ComponentBase
        where T : struct, IComparable
    {

        /// <summary>
        /// Initial value to set to the slider, data bound so it can also be read out
        /// </summary>
        [Parameter]
        public T Value { get; set; }

        public double ValueAsDouble { get; set; }

        public double GetValueAsDouble()
        {
            if (typeof(T).IsEnum)
            {
                if (_isInitialized)
                {
                    var e = _enumValues.FirstOrDefault(v => Convert.ToDouble(v).Equals(Convert.ToDouble(Value)));
                    return Convert.ToDouble(Convert.ChangeType(Value, typeof(int)));
                }
                else
                {
                    return 0;
                }
            }
            else
            {
                return Convert.ToDouble(Value);
            }
        }        

        [Parameter, EditorRequired]
        public required string Title { get; set; }

        [Parameter]
        public string? MinimumDescription { get; set; }

        [Parameter]
        public string? MaximumDescription { get; set; }

        [Parameter]
        public double Minimum { get; set; } = typeof(T).IsEnum ? Enum.GetValues(typeof(T)).Cast<int>().Select(e => Convert.ToDouble(e)).Min() : 0.0;

        [Parameter]
        public double Maximum { get; set; } = typeof(T).IsEnum ? Enum.GetValues(typeof(T)).Cast<int>().Select(e => Convert.ToDouble(e)).Max() : 100.0;

        [Parameter]
        public double? Stepsize { get; set; } = typeof(T).IsEnum ? 1 : 5.0;

        [Parameter]
        public bool ShowTickmarks { get; set; } = true;

        [Parameter]
        public AlternateStyle UseAlternateStyle { get; set; } = AlternateStyle.None;

        [Parameter]
        public EventCallback<T> ValueChanged { get; set; }

        public List<double> Tickmarks { get; set; } = new List<double>();

        private List<T> _enumValues { get; set; } = new List<T>();

        private bool _isInitialized = false;

        private async Task OnValueChanged(ChangeEventArgs e)
        {
            if (e.Value == null)
            {
                return;
            }
            if (typeof(T).IsEnum && e.Value != null)
            {
                var enumValue = _enumValues.FirstOrDefault(v => Convert.ToDouble(v).Equals(Convert.ToDouble(e.Value))); 
                if (Enum.TryParse(typeof(T), enumValue.ToString(), out _)) {
                    Value = enumValue; //check that it was a non-null value set from the slider
                }
                else
                {
                    return; //if we cannot handle the enum value set, do not process further
                }
            }
            else
            {
                Value = (T)Convert.ChangeType(e.Value!, typeof(T));
            }

            ValueAsDouble = GetValueAsDouble();

            await ValueChanged.InvokeAsync(Value);
        }


        private string TickmarksId = "ticksmarks_" + Guid.NewGuid().ToString("N");

        protected override async Task OnParametersSetAsync()
        {
            if (_isInitialized)
            {
                return ; //initialize ONCE 
            }

            if (!typeof(T).IsEnum && Value.CompareTo(0) == 0)
            {
                Value = (T)Convert.ChangeType((Convert.ToDouble(Maximum) - Convert.ToDouble(Minimum)) / 2, typeof(T));
                ValueAsDouble = GetValueAsDouble();
            }

            if (Maximum.CompareTo(Minimum) < 1)
            {
                throw new ArgumentException("The value for parameter 'Maximum' is set to a smaller value than {Minimum}");
            }
            GenerateTickMarks();

            BuildEnumValuesListIfRequired();

            _isInitialized = true;

            await Task.CompletedTask;
        }

        private void BuildEnumValuesListIfRequired()
        {
            if (typeof(T).IsEnum)
            {
                foreach (var item in Enum.GetValues(typeof(T)))
                {
                    _enumValues.Add((T)item);
                }
            }
        }

        private void GenerateTickMarks()
        {
            Tickmarks.Clear();
            if (!ShowTickmarks)
            {
                return;
            }
            if (typeof(T).IsEnum)
            {
                int enumValuesCount = Enum.GetValues(typeof(T)).Length;
                double offsetEnum = 0;
                double minDoubleValue = Enum.GetValues(typeof(T)).Cast<int>().Select(e => Convert.ToDouble(e)).Min();
                double maxDoubleValue = Enum.GetValues(typeof(T)).Cast<int>().Select(e => Convert.ToDouble(e)).Max();
                double enumStepSizeCalculated = (maxDoubleValue - minDoubleValue) / enumValuesCount;

                foreach (var enumValue in Enum.GetValues(typeof(T)))
                {
                    Tickmarks.Add(offsetEnum);
                    offsetEnum += Math.Round(enumStepSizeCalculated, 0);
                }
                return;
            }

            for (double i = Convert.ToDouble(Minimum); i <= Convert.ToDouble(Maximum); i += Convert.ToDouble(Stepsize))
            {
                Tickmarks.Add(i);
            }

        }      

    }

    public enum AlternateStyle
    {
        /// <summary>
        /// No alternate style. Uses the ordinary styling for the slider (browser default of input type 'range')
        /// </summary>
        None,

        /// <summary>
        /// Applies alternate style, using in addition to the 'slider track' an additional visual hint with an additional 'slider track' right below that shows a reddish color for lowest parts of the scale to the slider and towards yellow and greenish hues for higher values
        /// The alternate style uses a larger 'slider thumb' and alternate style to the 'slider-track'. The alternate style gives a more interesting look, especially in Microsoft Edge Chromium.
        /// </summary>
        AlternateStyle,

        /// <summary>
        /// Similar in style to the alternate style, but uses the inverse scale for the colors along the slider
        /// </summary>
        AlternateStyleInverseColorScale
    }

}


The markup of the Slider looks like this:


@using Microsoft.AspNetCore.Components.Forms
@using BlazorSliderLib
@typeparam T where T : struct, IComparable

<div class="slider-container sliderv2 @((UseAlternateStyle == AlternateStyle.AlternateStyle || (UseAlternateStyle == AlternateStyle.AlternateStyleInverseColorScale))? "alternate-style" : "") @(UseAlternateStyle == AlternateStyle.AlternateStyleInverseColorScale ? "alternate-style-inverse-colorscale" : "")">
<input type="range" @bind="@ValueAsDouble" min="@Minimum" max="@Maximum" step="@Stepsize" list="@TickmarksId" @oninput="OnValueChanged" />
<datalist id="@TickmarksId">
    @{
        var itemIndex = 0;
    }
    @foreach (var value in Tickmarks){
        if (typeof(T).IsEnum){
            var itemLabel = _enumValues.ElementAt(itemIndex);
            <option class="tick-label" value="@value" label="@itemLabel"></option>
        }
        else {
            <option class="tick-label" value="@value" label="@value"></option>
        }
        itemIndex++;    
    }
</datalist>

<div class="row">
@if (!string.IsNullOrWhiteSpace(MinimumDescription)){
    <div class="col-md-4">
        <label class="sliderv2Label text-muted">@MinimumDescription</label>
    </div>
}
@if (!string.IsNullOrWhiteSpace(Title)){
    <div class="col-md-4">
        <label class="sliderv2Label text-muted" style="text-align:center">@Title: @Value</label>
    </div>
}

@if (!string.IsNullOrWhiteSpace(MaximumDescription)){
    <div class="col-md-4" style="text-align:right">
        <label class="sliderv2Label tet-muted text-end">@MaximumDescription</label>
    </div>
}

</div>

<link rel="stylesheet" href="_content/BlazorSliderLib/Slider.css" />
<link rel="stylesheet" href="_content/BlazorSliderLib/SliderAlternate.css" />

<link rel="shortcut icon" type="image/x-icon" href="favicon.ico"/>


</div>


The slider control is provided "as is" and is free to change and use of no charge.

https://github.com/toreaurstadboss/BlazorSlider?tab=MIT-1-ov-file#readme

Monday, 22 April 2024

Pii - Detecting Personally Identifiable Information using Azure Cognitive Services

This article will look at detecting Person Identifiable Information (Pii) using Azure Cognitive Services. I have created a demo using .NET Maui Blazor has been created and the Github repo is here:
https://github.com/toreaurstadboss/PiiDetectionDemo

Person Identifiable Information (Pii) is desired to detect and also redact, that is using censorship or obscuring Pii to prepare documents for publication. The Pii feature in Azure Cognitive Services is a part of the Language resource service. A quickstart for using Pii is available here:
https://learn.microsoft.com/en-us/azure/ai-services/language-service/personally-identifiable-information/quickstart?pivots=programming-language-csharp

After creating the Language resource, look up the keys and endpoints for you service. Using Azure CLI inside Cloud shell, you can enter this command to find the keys, in Azure many services has got two keys you can exchange with new keys through regeneration:

az cognitiveservices account keys list --resource-group SomeAzureResourceGroup --name SomeAccountAzureCognitiveServices
This is how you can query after endpoint of language resource using Azure CLI : az cognitiveservices account show --query "properties.endpoint" --resource-group SomeAzureResourceGroup --name SomeAccountAzureCognitiveServices
Next, the demo of this article. Connecting to the Pii Removal Text Analytics is possible using this Nuget package (REST calls can also be done manually): - Azure.AI.TextAnalytics version 5.3.0 Here is the other Nugets of my Demo included from the .csproj file :

PiiDetectionDemo.csproj


  <ItemGroup>
        <PackageReference Include="Azure.AI.TextAnalytics" Version="5.3.0" />
        <PackageReference Include="Microsoft.Maui.Controls" Version="$(MauiVersion)" />
        <PackageReference Include="Microsoft.Maui.Controls.Compatibility" Version="$(MauiVersion)" />
        <PackageReference Include="Microsoft.AspNetCore.Components.WebView.Maui" Version="$(MauiVersion)" />
        <PackageReference Include="Microsoft.Extensions.Logging.Debug" Version="8.0.0" />
    </ItemGroup>


A service using this Pii removal feature is simply making use of a TextAnalyticsClient and method RecognizePiiEntitiesAsync.

PiiRemovalTextClientService.cs IPiiRemovalTextClientService.cs



using Azure;
using Azure.AI.TextAnalytics;

namespace PiiDetectionDemo.Util
{
    public interface IPiiRemovalTextAnalyticsClientService
    {
        Task<Response<PiiEntityCollection>> RecognizePiiEntitiesAsync(string? document, string? language);
    }
}


namespace PiiDetectionDemo.Util
{
    public class PiiRemovalTextAnalyticsClientService : IPiiRemovalTextAnalyticsClientService
    {

        private TextAnalyticsClient _client;

        public PiiRemovalTextAnalyticsClientService()
        {
            var azureEndpoint = Environment.GetEnvironmentVariable("AZURE_COGNITIVE_SERVICE_ENDPOINT");
            var azureKey = Environment.GetEnvironmentVariable("AZURE_COGNITIVE_SERVICE_KEY");

            if (string.IsNullOrWhiteSpace(azureEndpoint))
            {
                throw new ArgumentNullException(nameof(azureEndpoint), "Missing system environment variable: AZURE_COGNITIVE_SERVICE_ENDPOINT");
            }
            if (string.IsNullOrWhiteSpace(azureKey))
            {
                throw new ArgumentNullException(nameof(azureKey), "Missing system environment variable: AZURE_COGNITIVE_SERVICE_KEY");
            }

            _client = new TextAnalyticsClient(new Uri(azureEndpoint), new AzureKeyCredential(azureKey));
        }

        public async Task<Response<PiiEntityCollection>> RecognizePiiEntitiesAsync(string? document, string? language)
        {
            var piiEntities = await _client.RecognizePiiEntitiesAsync(document, language);
            return piiEntities;
        }

    }
}


The UI codebehind of the razor component page showing the UI looks like this:

Home.razor.cs


using Azure;
using Microsoft.AspNetCore.Components;
using PiiDetectionDemo.Models;
using PiiDetectionDemo.Util;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace PiiDetectionDemo.Components.Pages
{
    public partial class Home
    {

        private IndexModel Model = new();
        private bool isProcessing = false;
        private bool isSearchPerformed = false;

        private async Task Submit()
        {
            isSearchPerformed = false;
            isProcessing = true;
            try
            {
                var response = await _piiRemovalTextAnalyticsClientService.RecognizePiiEntitiesAsync(Model.InputText, null);
                Model.RedactedText = response?.Value?.RedactedText;
                Model.UpdateHtmlRedactedText();
                Model.AnalysisResult = response?.Value;
                StateHasChanged();
            }
            catch (Exception ex)
            {
                await Console.Out.WriteLineAsync(ex.ToString());
            }
            isProcessing = false;
            isSearchPerformed = true;
        }

        private void removeWhitespace(ChangeEventArgs args)
        {
            Model.InputText = args.Value?.ToString()?.CleanupAllWhiteSpace();
            StateHasChanged();
        }



    }
}



To get the redacted or censored text void of any Pii that the Pii detection feature was able to detect, access the Value of type Azure.AI.TextAnalytics.PiiEntityCollection. Inside this object, the string RedactedText contains the censored / redacted text. The IndexModel looks like this :


using Azure.AI.TextAnalytics;
using Microsoft.AspNetCore.Components;
using PiiDetectionDemo.Util;
using System.ComponentModel.DataAnnotations;
using System.Text;

namespace PiiDetectionDemo.Models
{

    public class IndexModel
    {

        [Required]
        public string? InputText { get; set; }

        public string? RedactedText { get; set; }

        public string? HtmlRedactedText { get; set; }

        public MarkupString HtmlRedactedTextMarkupString { get; set; }

        public void UpdateHtmlRedactedText()
        {
            var sb = new StringBuilder(RedactedText);
            if (AnalysisResult != null && RedactedText != null)
            {
                foreach (var piiEntity in AnalysisResult.OrderByDescending(a => a.Offset))
                {
                    sb.Insert(piiEntity.Offset + piiEntity.Length, "</b></span>");
                    sb.Insert(piiEntity.Offset, $"<span style='background-color:lightgray;border:1px solid black;corner-radius:2px; color:{GetBackgroundColor(piiEntity)}' title='{piiEntity.Category}: {piiEntity.SubCategory} Confidence: {piiEntity.ConfidenceScore} Redacted Text: {piiEntity.Text}'><b>");
                }
            }
            HtmlRedactedText = sb.ToString()?.CleanupAllWhiteSpace();    
            HtmlRedactedTextMarkupString = new MarkupString(HtmlRedactedText ?? string.Empty);
        }

        private string GetBackgroundColor(PiiEntity piiEntity)
        {
            if (piiEntity.Category == PiiEntityCategory.PhoneNumber)
            {
                return "yellow";
            }
            if (piiEntity.Category == PiiEntityCategory.Organization)
            {
                return "orange";
            }
            if (piiEntity.Category == PiiEntityCategory.Address)
            {
                return "green";
            }
            return "gray";                   
        }

        public long ExecutionTime { get; set; }
        public PiiEntityCollection? AnalysisResult { get; set; }

    }
}




Frontend UI looks like this: Home.razor


@page "/"
@using PiiDetectionDemo.Util

@inject IPiiRemovalTextAnalyticsClientService _piiRemovalTextAnalyticsClientService;

<h3>Azure HealthCare Text Analysis - Pii detection feature - Azure Cognitive Services</h3>

<em>Pii = Person identifiable information</em>

<EditForm Model="@Model" OnValidSubmit="@Submit">
    <DataAnnotationsValidator />
    <ValidationSummary />

    <div class="form-group row">
        <label><strong>Text input</strong></label>
        <InputTextArea @oninput="removeWhitespace" class="overflow-scroll" style="max-height:500px;max-width:900px;font-size: 10pt;font-family:Verdana, Geneva, Tahoma, sans-serif" @bind-Value="@Model.InputText" rows="5" />
    </div>

    <div class="form-group row">
        <div class="col">
            <br />
            <button class="btn btn-outline-primary" type="submit">Run</button>
        </div>
        <div class="col">
        </div>
        <div class="col">
        </div>
    </div>

    <br />

    @if (isProcessing)
    {

        <div class="progress" style="max-width: 90%">
            <div class="progress-bar progress-bar-striped progress-bar-animated"
                 style="width: 100%; background-color: green">
                Retrieving result from Azure Text Analysis Pii detection feature. Processing..
            </div>
        </div>
        <br />

    }

    <div class="form-group row">
        <label><strong>Analysis result</strong></label>

        @if (isSearchPerformed)
        {
            <br />
            <b>Execution time took: @Model.ExecutionTime ms (milliseconds)</b>

            <br />
            <br />

            <b>Redacted text (Pii removed)</b>
            <br />

            <div class="form-group row">
               <label><strong>Categorized Pii redacted text</strong></label>
               <div>
               @Model.HtmlRedactedTextMarkupString
               </div>
            </div>

            <br />
            <br />

            <table class="table table-striped table-dark table-hover">
                <thead>
                <th>Pii text</th>
                <th>Category</th>
                <th>SubCategory</th>
                <th>Offset</th>
                <th>Length</th>
                <th>ConfidenceScore</th>
                </thead>
                <tbody>
                    @if (Model.AnalysisResult != null) {
                        @foreach (var entity in Model.AnalysisResult)
                        {
                            <tr>
                                <td>@entity.Text</td>
                                <td>@entity.Category.ToString()</td>
                                <td>@entity.SubCategory</td>
                                <td>@entity.Offset</td>
                                <td>@entity.Length</td>
                                <td>@entity.ConfidenceScore</td>                                        
                            </tr>
                        }
                    }
                </tbody>
            </table>

        }
    </div>

</EditForm>



The Demo uses Bootstrap 5 to build up a HTML table styled and showing the Azure.AI.TextAnalytics.PiiEntity properties.

Saturday, 21 October 2023

Using Azure Health Information extraction in Azure Cognitive Services

This article presents code how to extract Health information from arbitrary text using Azure Health Information extraction in Azure Cognitive Services. This technology uses NLP - natural language processing combined with AI techniques. A Github repo exists with the code for a running .NET MAUI Blazor demo in .NET 7 here:

https://github.com/toreaurstadboss/HealthTextAnalytics

A screenshot from the demo shows how it works below. The demo uses Azure AI Healthcare information extraction to extract entities of the text, such as a person's age, gender, employment and medical history and condition such as diagnosises, procedures and so on. The returned data in the demo is shown at the bottom of the demo, the raw data shows it is in the format as a json and in a FHIR format. Since we want FHIR format, we must use the REST api to get this information. Azure AI Healthcare information also extracts relations, which is connecting the entities together for semantic analysis of the text. Also, links exist for each entity for further reading. These are external systems such as Snomed CT and Snomed codes for each entity. Let's look at the source code for the demo next. We define a named http client in the MauiProgram.cs file which starts the application. We could move the code into a middleware extension method, but the code is kept simple in the demo.

MauiProgram.cs


  var azureEndpoint = Environment.GetEnvironmentVariable("AZURE_COGNITIVE_SERVICES_LANGUAGE_SERVICE_ENDPOINT");
  var azureKey = Environment.GetEnvironmentVariable("AZURE_COGNITIVE_SERVICES_LANGUAGE_SERVICE_KEY");

  if (string.IsNullOrWhiteSpace(azureEndpoint))
  {
      throw new ArgumentNullException(nameof(azureEndpoint), "Missing system environment variable: AZURE_COGNITIVE_SERVICES_LANGUAGE_SERVICE_ENDPOINT");
  }
  if (string.IsNullOrWhiteSpace(azureKey))
  {
      throw new ArgumentNullException(nameof(azureKey), "Missing system environment variable: AZURE_COGNITIVE_SERVICES_LANGUAGE_SERVICE_KEY");
  }

  var azureEndpointHost = new Uri(azureEndpoint);

  builder.Services.AddHttpClient("Az", httpClient =>
  {
      string baseUrl = azureEndpointHost.GetLeftPart(UriPartial.Authority); //https://stackoverflow.com/a/18708268/741368
      httpClient.BaseAddress = new Uri(baseUrl);
      //httpClient..Add("Content-type", "application/json");
      //httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));//ACCEPT header
      httpClient.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", azureKey);
  });



The content-type header will be specified instead inside the HttpRequestMessage shown further below and not in this named client. As we see, we must add both the endpoint base url and also the key in the Ocp-Apim-Subscription-key http header. Let's next look at how to create a POST request to the language resource endpoint that offers the health text analysis below.

HealthAnalyticsTextClientService.cs




using HealthTextAnalytics.Models;
using System.Diagnostics;
using System.Text;
using System.Text.Json.Nodes;

namespace HealthTextAnalytics.Util
{

    public class HealthAnalyticsTextClientService : IHealthAnalyticsTextClientService
    {

        private readonly IHttpClientFactory _httpClientFactory;
        private const int awaitTimeInMs = 500;
        private const int maxTimerWait = 10000;

        public HealthAnalyticsTextClientService(IHttpClientFactory httpClientFactory)
        {
            _httpClientFactory = httpClientFactory;
        }

        public async Task<HealthTextAnalyticsResponse> GetHealthTextAnalytics(string inputText)
        {
            var client = _httpClientFactory.CreateClient("Az");
            string requestBodyRaw = HealthAnalyticsTextHelper.CreateRequest(inputText);
            //https://learn.microsoft.com/en-us/azure/ai-services/language-service/text-analytics-for-health/how-to/call-api?tabs=ner
            var stopWatch = Stopwatch.StartNew();
            HttpRequestMessage request = CreateTextAnalyticsRequest(requestBodyRaw);
            var response = await client.SendAsync(request);
            var result = new HealthTextAnalyticsResponse();
            var timer = new PeriodicTimer(TimeSpan.FromMilliseconds(awaitTimeInMs));
            int timeAwaited = 0;

            while (await timer.WaitForNextTickAsync())
            {
                if (response.IsSuccessStatusCode)
                {
                    result.IsSearchPerformed = true;
                    var operationLocation = response.Headers.First(h => h.Key?.ToLower() == Constants.Constants.HttpHeaderOperationResultAvailable).Value.FirstOrDefault();

                    var resultFromHealthAnalysis = await client.GetAsync(operationLocation);
                    JsonNode resultFromService = await resultFromHealthAnalysis.GetJsonFromHttpResponse();
                    if (resultFromService.GetValue<string>("status") == "succeeded")
                    {
                        result.AnalysisResultRawJson = await resultFromHealthAnalysis.Content.ReadAsStringAsync();
                        result.ExecutionTimeInMilliseconds = stopWatch.ElapsedMilliseconds;
                        result.Entities.AddRange(HealthAnalyticsTextHelper.GetEntities(result.AnalysisResultRawJson));
                        result.CategorizedInputText = HealthAnalyticsTextHelper.GetCategorizedInputText(inputText, result.AnalysisResultRawJson);
                        break;
                    }
                }
                timeAwaited += 500;
                if (timeAwaited >= maxTimerWait)
                {
                    result.CategorizedInputText = $"ERR: Timeout. Operation to analyze input text using Azure HealthAnalytics language service timed out after waiting for {timeAwaited} ms.";
                    break;
                }
            }

            return result;
        }

        private static HttpRequestMessage CreateTextAnalyticsRequest(string requestBodyRaw)
        {
            var request = new HttpRequestMessage(HttpMethod.Post, Constants.Constants.AnalyzeTextEndpoint);
            request.Content = new StringContent(requestBodyRaw, Encoding.UTF8, "application/json");//CONTENT-TYPE header
            return request;
        }
    }

}



The code is using some helper methods to be shown next. As the code above shows, we must poll the Azure service until we get a reply from the service. We poll every 0.5 second up to a maxium of 10 seconds from the service. Typical requests takes about 3-4 seconds to process. Longer input text / 'documents' would need more processing time than 10 seconds, but for this demo, it works great.

HealthAnalyticsTextHelper.CreateRequest method


  public static string CreateRequest(string inputText)
  {
      //note - the id 1 here in the request is a 'local id' that must be unique per request. only one text is supported in the 
      //request genreated, however the service allows multiple documents and id's if necessary. in this demo, we only will send in one text at a time
      var request = new
      {
          analysisInput = new
          {
              documents = new[]
              {
                  new { text = inputText, id = "1", language = "en" }
              }
          },
          tasks = new[]
          {
              new { id = "analyze 1", kind = "Healthcare", parameters = new { fhirVersion = "4.0.1" } }
          }
      };
      return JsonSerializer.Serialize(request, new JsonSerializerOptions { WriteIndented = true });
  }



Creating the body of POST we use a template via a new anonymized object shown above which is what the REST service excepts. We could have multiple documents here, that is input texts, in this demo only one text / document is sent in. Note the use of id='1' and 'analyze 1' here. We have some helper methods in System.Text.Json here to extract the JSON data sent in the response.

JsonNodeUtil


 public static class JsonNodeUtil
 {

     public static async Task<JsonNode> GetJsonFromHttpResponse(this HttpResponseMessage response)
     {
         var resultFromService = JsonSerializer.Deserialize<JsonNode>(await response.Content.ReadAsStringAsync());
         return resultFromService;
     }

     public static T? GetValue<T>(this JsonNode jsonNode, string key)
     {
         if (jsonNode == null)
         {
             return default;
         }
         return jsonNode[key] != null ? jsonNode[key].GetValue<T>() : default;
     }

 }

 

More code exists for returning a categorized colored input text showing the entities of the input text in the helper below.

HealthAnalyticsTextHelper.cs - methods GetCategorizedInputText and GetBackgroundColor


 public static string GetCategorizedInputText(string inputText, string analysisText)
 {
     var sb = new StringBuilder(inputText);
     try
     {
         Root doc = JsonSerializer.Deserialize<Root>(analysisText);

         //try loading up the documents inside of the analysisText
         var entities = doc?.tasks?.items.FirstOrDefault()?.results?.documents?.SelectMany(d => d.entities)?.ToList();
         if (entities != null)
         {
             foreach (var row in entities.OrderByDescending(r => r.offset))
             {
                 sb.Insert(row.offset + row.length, "</b></span>");
                 sb.Insert(row.offset, $"<span style='color:{GetBackgroundColor(row)}' title='{row.category}: {row.text} Confidence: {row.confidenceScore} {row.name}'><b>");
             }
         }
     }
     catch (Exception err)
     {

         Console.WriteLine("Got an error while trying to load in analysis healthcare json: " + err.ToString());
     }
     return $"<pre style='text-wrap:wrap; max-height:500px;font-size: 10pt;font-family:Verdana, Geneva, Tahoma, sans-serif;'>{sb}</pre>";
 }

 private static string GetBackgroundColor(Entity row)
 {
     var cat = row?.category?.ToLower();
     string backgroundColor = cat switch
     {
         "age" => "purple",
         "diagnosis" => "orange",
         "gender" => "purple",
         "symptomorsign" => "purple",
         "direction" => "blue",
         "symptom" => "purple",
         "symptoms" => "purple",
         "bodystructure" => "blue",
         "body" => "purple",
         "structure" => "purple",
         "examinationname" => "green",
         "procedure" => "green",
         "treatmentname" => "green",
         "conditionqualifier" => "lightgreen",
         "time" => "lightgreen",
         "date" => "lightgreen",
         "familyrelation" => "purple",
         "employment" => "purple",
         "livingstatus" => "purple",
         "administrativeevent" => "darkgreen",
         "careenvironment" => "darkgreen",
         _ => "darkgray"
     };
     return backgroundColor;
 }




I have added the Domain classes from the service using the https://json2csharp.com/ website on the intial responses I got from the REST service using Postman. The REST Api might change in the future, that is, the JSON returned. In that case, you might want to adjust the domain classes here if the deserialization fails. It seems relatively stable though, I have tested the code for some weeks now. Finally, the categorized colored text code here had to remove newlines to get a correct indexing of the different entities found in the text. This code shows how to get rid of newlines of the inputted text.


 public static class StringExtensions
 {
     
     public static string CleanupAllWhiteSpace(this string input) => Regex.Replace(input ?? string.Empty, @"\s+", " ");
     
 }


Let's look at the UI in the Index.razor file below.

Index.razor


@page "/"
@using HealthTextAnalytics.Models;
@inject IHttpClientFactory _httpClientFactory;
@inject IHealthAnalyticsTextClientService _healthAnalyticsTextClientService;

<h3>Azure HealthCare Text Analysis - Azure Cognitive Services</h3>

<EditForm Model="@Model" OnValidSubmit="@Submit">
    <DataAnnotationsValidator />
    <ValidationSummary />

    <InputWatcher @ref="inputWatcher" FieldChanged="@FieldChanged" />

    <div class="form-group row">
        <label><strong>Text input</strong></label>
        <InputTextArea @onkeyup="@removeWhitespace" class="overflow-scroll" style="max-height:500px;max-width:900px;font-size: 10pt;font-family:Verdana, Geneva, Tahoma, sans-serif" @bind-Value="@Model.InputText" rows="5" />
    </div>

    <div class="form-group row"> 
        <div class="col">
            <br />
            <button class="btn btn-outline-primary" disabled="@isInvalid" type="submit">Run</button>
        </div>
        <div class="col">
        </div>
        <div class="col">
        </div>
    </div>

    <br />

@if (isProcessing)
{

        <div class="progress" style="max-width: 90%">
            <div class="progress-bar progress-bar-striped progress-bar-animated"
                 style="width: 100%; background-color: green">
                 Retrieving result from Azure HealthCare Text Analysis. Processing..
            </div>
        </div>
        <br />

}

    <div class="form-group row">
        <label><strong>Analysis result</strong></label>

        @if (isSearchPerformed)
    {
        <br />
        <b>Execution time took: @Model.ExecutionTime ms (milliseconds)</b><br />
        <br />

        <b>Categorized and analyzed Health Analysis of inputted text</b>
        @ms
        <br />
     
        <table class="table table-striped table-dark table-hover">
                <th>Category</th>
                <th>Text</th>
                <th>Name</th>
                <th>ConfidenceScore</th>
                <th>Offset</th>
                <th>Length</th>
                <th>Links</th>
            <tbody>
            @foreach (var entity in Model.EntititesInAnalyzedResult)
        {
            <tr>
                    <td>@entity.category</td>
                    <td>@entity.text</td>
                    <td>@entity.name</td>
                    <td>@entity.confidenceScore</td>
                    <td>@entity.offset</td>
                    <td>@entity.length</td>
                    <td>@string.Join(Environment.NewLine, (@entity.links ?? new List<Link>()).Select(l => l?.dataSource + " " + l?.id + " | "))</td>
                </tr>
            
        }
            </tbody>
            </table>

        <b>Health Analysis raw text from Azure service</b>
        <InputTextArea class="overflow-scroll" readonly="readonly" style="max-height:500px; max-width:900px;font-size: 10pt;font-family:Verdana, Geneva, Tahoma, sans-serif" @bind-Value="@Model.AnalysisResult" rows="1000" />

    }
   </div>

</EditForm>


The code-behind of Index.razor , looks like this.


using HealthTextAnalytics.Models;
using HealthTextAnalytics.Util;
using Microsoft.AspNetCore.Components;
using Microsoft.AspNetCore.Components.Web;

namespace HealthTextAnalytics.Pages
{
    public partial class Index
    {

        private IndexModel Model = new();
        MarkupString ms = new();
        private bool isProcessing = false;
        private bool isSearchPerformed = false;   

        private InputWatcher inputWatcher = new InputWatcher();
        private bool isInvalid = false;

        private void FieldChanged(string fieldName)
        {
            isInvalid = !inputWatcher.Validate();
        }
        
        protected override void OnParametersSet()
        {
            Model.InputText = SampleData.Sampledata.SamplePatientTextNote2.CleanupAllWhiteSpace();
            StateHasChanged();
        }

        private void removeWhitespace(KeyboardEventArgs eventArgs)
        {
            Model.InputText = Model.InputText.CleanupAllWhiteSpace();
            StateHasChanged();
        }

        private async Task Submit()
        {
            try
            {
                ResetFieldsForBeforeSearch();

                HealthTextAnalyticsResponse response = await _healthAnalyticsTextClientService.GetHealthTextAnalytics(Model.InputText);
                Model.EntititesInAnalyzedResult = response.Entities;
                Model.ExecutionTime = response.ExecutionTimeInMilliseconds;
                Model.AnalysisResult = response.AnalysisResultRawJson;

                ms = new MarkupString(response.CategorizedInputText);              
            }
            catch (Exception err)
            {
                Console.WriteLine(err);
            }
            finally
            {
                ResetFieldsAfterSearch();
                StateHasChanged();
            }
        }

        private void ResetFieldsForBeforeSearch()
        {
            isProcessing = true;
            isSearchPerformed = false;
            ms = new MarkupString(string.Empty);
            Model.EntititesInAnalyzedResult.Clear();
            Model.AnalysisResult = string.Empty;
        }

        private void ResetFieldsAfterSearch()
        {
            isProcessing = false;
            isSearchPerformed = true;
        }

    }
}


Saturday, 14 October 2023

Using Image Analysis in Azure AI Cognitive Services

I have added a demo .NET MAUI Blazor app that uses Image Analysis in Computer Vision in Azure Cognitive Services. Note that Image Analysis is not available in all Azure data centers. For example, Norway East does not have this feature. However, North Europe Azure data center do have the feature, the data center i Ireland. A Github repo exists for this demo here:

https://github.com/toreaurstadboss/Image.Analyze.Azure.Ai

A screen shot for this demo is shown below: Demo screenshot The demo allows you to upload a picture (supported formats are .jpeg, .jpg and .png, but Azure AI Image Analyzer supports a lot of other image formats too). The demo shows a preview of the selected image and to the right an image of bounding boxes of objects in the image. A list of tags extracted from the image are also shown. Raw data from the Azure Image Analyzer service is shown in the text box area below the pictures, with a list of tags to the right. The demo is written with .NET Maui Blazor and .NET 6. Let us look at some code for making this demo. ImageSaveService.cs


using Image.Analyze.Azure.Ai.Models;
using Microsoft.AspNetCore.Components.Forms;

namespace Ocr.Handwriting.Azure.AI.Services
{

    public class ImageSaveService : IImageSaveService
    {

        public async Task<ImageSaveModel> SaveImage(IBrowserFile browserFile)
        {
            var buffers = new byte[browserFile.Size];
            var bytes = await browserFile.OpenReadStream(maxAllowedSize: 30 * 1024 * 1024).ReadAsync(buffers);
            string imageType = browserFile.ContentType;

            var basePath = FileSystem.Current.AppDataDirectory;
            var imageSaveModel = new ImageSaveModel
            {
                SavedFilePath = Path.Combine(basePath, $"{Guid.NewGuid().ToString("N")}-{browserFile.Name}"),
                PreviewImageUrl = $"data:{imageType};base64,{Convert.ToBase64String(buffers)}",
                FilePath = browserFile.Name,
                FileSize = bytes / 1024,
            };

            await File.WriteAllBytesAsync(imageSaveModel.SavedFilePath, buffers);

            return imageSaveModel;
        }

    }
}

//Interface defined inside IImageService.cs shown below
using Image.Analyze.Azure.Ai.Models;
using Microsoft.AspNetCore.Components.Forms;

namespace Ocr.Handwriting.Azure.AI.Services
{
  
    public interface IImageSaveService
    {

        Task<ImageSaveModel> SaveImage(IBrowserFile browserFile);

    }

}


The ImageSaveService saves the uploaded image from the IBrowserFile into a base-64 string from the image bytes of the uploaded IBrowserFile via OpenReadStream of the IBrowserFile. This allows us to preview the uploaded image. The code also saves the image to the AppDataDirectory that MAUI supports - FileSystem.Current.AppDataDirectory. Let's look at how to call the analysis service itself, it is actually quite straight forward. ImageAnalyzerService.cs


using Azure;
using Azure.AI.Vision.Common;
using Azure.AI.Vision.ImageAnalysis;

namespace Image.Analyze.Azure.Ai.Lib
{

    public class ImageAnalyzerService : IImageAnalyzerService
    {

        public ImageAnalyzer CreateImageAnalyzer(string imageFile)
        {
            string key = Environment.GetEnvironmentVariable("AZURE_COGNITIVE_SERVICES_VISION_SECONDARY_KEY");
            string endpoint = Environment.GetEnvironmentVariable("AZURE_COGNITIVE_SERVICES_VISION_SECONDARY_ENDPOINT");
            var visionServiceOptions = new VisionServiceOptions(new Uri(endpoint), new AzureKeyCredential(key));

            using VisionSource visionSource = CreateVisionSource(imageFile);

            var analysisOptions = CreateImageAnalysisOptions();

            var analyzer = new ImageAnalyzer(visionServiceOptions, visionSource, analysisOptions);
            return analyzer;

        }

        private static VisionSource CreateVisionSource(string imageFile)
        {
            using var stream = File.OpenRead(imageFile);
            using var reader = new StreamReader(stream);
            byte[] imageBuffer;
            using (var streamReader = new MemoryStream())
            {
                stream.CopyTo(streamReader);
                imageBuffer = streamReader.ToArray();
            }

            using var imageSourceBuffer = new ImageSourceBuffer();
            imageSourceBuffer.GetWriter().Write(imageBuffer);
            return VisionSource.FromImageSourceBuffer(imageSourceBuffer);
        }

        private static ImageAnalysisOptions CreateImageAnalysisOptions() => new ImageAnalysisOptions
        {
            Language = "en",
            GenderNeutralCaption = false,
            Features =
              ImageAnalysisFeature.CropSuggestions
            | ImageAnalysisFeature.Caption
            | ImageAnalysisFeature.DenseCaptions
            | ImageAnalysisFeature.Objects
            | ImageAnalysisFeature.People
            | ImageAnalysisFeature.Text
            | ImageAnalysisFeature.Tags
        };

    }

}

//interface shown below 

 public interface IImageAnalyzerService
 {
     ImageAnalyzer CreateImageAnalyzer(string imageFile);
 }



We retrieve environment variables here and we create an ImageAnalyzer. We create a Vision source from the saved picture we uploaded and open a stream to it using File.OpenRead method on System.IO. Since we saved the file in the AppData folder of the .NET MAUI app, we can read this file. We set up the image analysis options and the vision service options. We then call the return the image analyzer. Let's look at the code-behind of the index.razor file that initializes the Image analyzer, and runs the Analyze method of it. Index.razor.cs
 
 
 using Azure.AI.Vision.ImageAnalysis;
using Image.Analyze.Azure.Ai.Extensions;
using Image.Analyze.Azure.Ai.Models;
using Microsoft.AspNetCore.Components.Forms;
using Microsoft.JSInterop;
using System.Text;

namespace Image.Analyze.Azure.Ai.Pages
{
    partial class Index
    {

        private IndexModel Model = new();

        //https://learn.microsoft.com/en-us/azure/ai-services/computer-vision/how-to/call-analyze-image-40?WT.mc_id=twitter&pivots=programming-language-csharp

        private string ImageInfo = string.Empty;

        private async Task Submit()
        {
            if (Model.PreviewImageUrl == null || Model.SavedFilePath == null)
            {
                await Application.Current.MainPage.DisplayAlert($"MAUI Blazor Image Analyzer App", $"You must select an image first before running Image Analysis. Supported formats are .jpeg, .jpg and .png", "Ok", "Cancel");
                return;
            }

            using var imageAnalyzer = ImageAnalyzerService.CreateImageAnalyzer(Model.SavedFilePath);

            ImageAnalysisResult analysisResult = await imageAnalyzer.AnalyzeAsync();

            if (analysisResult.Reason == ImageAnalysisResultReason.Analyzed)
            {
                Model.ImageAnalysisOutputText = analysisResult.OutputImageAnalysisResult();
                Model.Caption = $"{analysisResult.Caption.Content} Confidence: {analysisResult.Caption.Confidence.ToString("F2")}";
                Model.Tags = analysisResult.Tags.Select(t => $"{t.Name} (Confidence: {t.Confidence.ToString("F2")})").ToList();
                var jsonBboxes = analysisResult.GetBoundingBoxesJson();
                await JsRunTime.InvokeVoidAsync("LoadBoundingBoxes", jsonBboxes);
            }
            else
            {
                ImageInfo = $"The image analysis did not perform its analysis. Reason: {analysisResult.Reason}";
            }

            StateHasChanged(); //visual refresh here
        }

        private async Task CopyTextToClipboard()
        {
            await Clipboard.SetTextAsync(Model.ImageAnalysisOutputText);
            await Application.Current.MainPage.DisplayAlert($"MAUI Blazor Image Analyzer App", $"The copied text was put into the clipboard. Character length: {Model.ImageAnalysisOutputText?.Length}", "Ok", "Cancel");
        }

        private async Task OnInputFile(InputFileChangeEventArgs args)
        {
            var imageSaveModel = await ImageSaveService.SaveImage(args.File);
            Model = new IndexModel(imageSaveModel);
            await Application.Current.MainPage.DisplayAlert($"MAUI Blazor ImageAnalyzer app App", $"Wrote file to location : {Model.SavedFilePath} Size is: {Model.FileSize} kB", "Ok", "Cancel");
        }


    }
}
 
 
In the code-behind above we have a submit handler called Submit. We there analyze the image and send the result both to the UI and also to a client side Javascript method using IJSRuntime in .NET MAUI Blazor. Let's look at the two helper methods of ImageAnalysisResult next. ImageAnalysisResultExtensions.cs
 
 
 using Azure.AI.Vision.ImageAnalysis;
using System.Text;

namespace Image.Analyze.Azure.Ai.Extensions
{
    public static class ImageAnalysisResultExtensions
    {

        public static string GetBoundingBoxesJson(this ImageAnalysisResult result)
        {
            var sb = new StringBuilder();
            sb.AppendLine(@"[");

            int objectIndex = 0;
            foreach (var detectedObject in result.Objects)
            {
                sb.Append($"{{ \"Name\": \"{detectedObject.Name}\", \"Y\": {detectedObject.BoundingBox.Y}, \"X\": {detectedObject.BoundingBox.X}, \"Height\": {detectedObject.BoundingBox.Height}, \"Width\": {detectedObject.BoundingBox.Width}, \"Confidence\": \"{detectedObject.Confidence:0.0000}\" }}");
                objectIndex++;
                if (objectIndex < result.Objects?.Count)
                {
                    sb.Append($",{Environment.NewLine}");
                }
                else
                {
                    sb.Append($"{Environment.NewLine}");
                }
            }
            sb.Remove(sb.Length - 2, 1); //remove trailing comma at the end
            sb.AppendLine(@"]");
            return sb.ToString();
        }

        public static string OutputImageAnalysisResult(this ImageAnalysisResult result)
        {
            var sb = new StringBuilder();

            if (result.Reason == ImageAnalysisResultReason.Analyzed)
            {

                sb.AppendLine($" Image height = {result.ImageHeight}");
                sb.AppendLine($" Image width = {result.ImageWidth}");
                sb.AppendLine($" Model version = {result.ModelVersion}");

                if (result.Caption != null)
                {
                    sb.AppendLine(" Caption:");
                    sb.AppendLine($"   \"{result.Caption.Content}\", Confidence {result.Caption.Confidence:0.0000}");
                }

                if (result.DenseCaptions != null)
                {
                    sb.AppendLine(" Dense Captions:");
                    foreach (var caption in result.DenseCaptions)
                    {
                        sb.AppendLine($"   \"{caption.Content}\", Bounding box {caption.BoundingBox}, Confidence {caption.Confidence:0.0000}");
                    }
                }

                if (result.Objects != null)
                {
                    sb.AppendLine(" Objects:");
                    foreach (var detectedObject in result.Objects)
                    {
                        sb.AppendLine($"   \"{detectedObject.Name}\", Bounding box {detectedObject.BoundingBox}, Confidence {detectedObject.Confidence:0.0000}");
                    }
                }

                if (result.Tags != null)
                {
                    sb.AppendLine($" Tags:");
                    foreach (var tag in result.Tags)
                    {
                        sb.AppendLine($"   \"{tag.Name}\", Confidence {tag.Confidence:0.0000}");
                    }
                }

                if (result.People != null)
                {
                    sb.AppendLine($" People:");
                    foreach (var person in result.People)
                    {
                        sb.AppendLine($"   Bounding box {person.BoundingBox}, Confidence {person.Confidence:0.0000}");
                    }
                }

                if (result.CropSuggestions != null)
                {
                    sb.AppendLine($" Crop Suggestions:");
                    foreach (var cropSuggestion in result.CropSuggestions)
                    {
                        sb.AppendLine($"   Aspect ratio {cropSuggestion.AspectRatio}: "
                            + $"Crop suggestion {cropSuggestion.BoundingBox}");
                    };
                }

                if (result.Text != null)
                {
                    sb.AppendLine($" Text:");
                    foreach (var line in result.Text.Lines)
                    {
                        string pointsToString = "{" + string.Join(',', line.BoundingPolygon.Select(pointsToString => pointsToString.ToString())) + "}";
                        sb.AppendLine($"   Line: '{line.Content}', Bounding polygon {pointsToString}");

                        foreach (var word in line.Words)
                        {
                            pointsToString = "{" + string.Join(',', word.BoundingPolygon.Select(pointsToString => pointsToString.ToString())) + "}";
                            sb.AppendLine($"     Word: '{word.Content}', Bounding polygon {pointsToString}, Confidence {word.Confidence:0.0000}");
                        }
                    }
                }

                var resultDetails = ImageAnalysisResultDetails.FromResult(result);
                sb.AppendLine($" Result details:");
                sb.AppendLine($"   Image ID = {resultDetails.ImageId}");
                sb.AppendLine($"   Result ID = {resultDetails.ResultId}");
                sb.AppendLine($"   Connection URL = {resultDetails.ConnectionUrl}");
                sb.AppendLine($"   JSON result = {resultDetails.JsonResult}");
            }
            else
            {
                var errorDetails = ImageAnalysisErrorDetails.FromResult(result);
                sb.AppendLine(" Analysis failed.");
                sb.AppendLine($"   Error reason : {errorDetails.Reason}");
                sb.AppendLine($"   Error code : {errorDetails.ErrorCode}");
                sb.AppendLine($"   Error message: {errorDetails.Message}");
            }

            return sb.ToString();
        }

    }
}


  
 
Finally, let's look at the client side Javascript function that we call and send the bounding boxes json to draw the boxes. We will use Canvas in HTML 5 to show the picture and the bounding boxes of objects found in the image. index.html
 
 
 	<script type="text/javascript">

		var colorPalette = ["red", "yellow", "blue", "green", "fuchsia", "moccasin", "purple", "magenta", "aliceblue", "lightyellow", "lightgreen"];

		function rescaleCanvas() {
			var img = document.getElementById('PreviewImage');
			var canvas = document.getElementById('PreviewImageBbox');
			canvas.width = img.width;
			canvas.height = img.height;
		}

		function getColor() {
			var colorIndex = parseInt(Math.random() * 10);
			var color = colorPalette[colorIndex];
			return color;
		}

		function LoadBoundingBoxes(objectDescriptions) {
			if (objectDescriptions == null || objectDescriptions == false) {
				alert('did not find any objects in image. returning from calling load bounding boxes : ' + objectDescriptions);
				return;
			}

			var objectDesc = JSON.parse(objectDescriptions);
			//alert('calling load bounding boxes, starting analysis on clientside js : ' + objectDescriptions);

			rescaleCanvas();
			var canvas = document.getElementById('PreviewImageBbox');
			var img = document.getElementById('PreviewImage');

			var ctx = canvas.getContext('2d');
			ctx.drawImage(img, img.width, img.height);

			ctx.font = "10px Verdana";

			for (var i = 0; i < objectDesc.length; i++) {
				ctx.beginPath();
				ctx.strokeStyle = "black";
				ctx.lineWidth = 1;
				ctx.fillText(objectDesc[i].Name, objectDesc[i].X + objectDesc[i].Width / 2, objectDesc[i].Y + objectDesc[i].Height / 2);
				ctx.fillText("Confidence: " + objectDesc[i].Confidence, objectDesc[i].X + objectDesc[i].Width / 2, 10 + objectDesc[i].Y + objectDesc[i].Height / 2);
			}

			for (var i = 0; i < objectDesc.length; i++) {
				ctx.fillStyle = getColor();
				ctx.globalAlpha = 0.2;
				ctx.fillRect(objectDesc[i].X, objectDesc[i].Y, objectDesc[i].Width, objectDesc[i].Height);
				ctx.lineWidth = 3;
				ctx.strokeStyle = "blue";
				ctx.rect(objectDesc[i].X, objectDesc[i].Y, objectDesc[i].Width, objectDesc[i].Height);
				ctx.fillStyle = "black";
				ctx.fillText("Color: " + getColor(), objectDesc[i].X + objectDesc[i].Width / 2, 20 + objectDesc[i].Y + objectDesc[i].Height / 2);

				ctx.stroke();
			}

			ctx.drawImage(img, 0, 0);


			console.log('got these object descriptions:');
			console.log(objectDescriptions);

		}
	</script>

 
  
The index.html file in wwwroot is the place we usually put extra css and js for MAUI Blazor apps and Blazor apps. I have chosen to put the script directly into the index.html file and not in a .js file, but that is an option to be chosen to tidy up a bit more. So there you have it, we can relatively easily find objects in images using Azure analyze image service in Azure Cognitive Services. We can get tags and captions of the image. In the demo the caption is shown above the picture loaded. Azure Computer vision service is really good since it has got a massive training set and can recognize a lot of different objects for different usages. As you see in the source code, I have the key and endpoint inside environment variables that the code expects exists. Never expose keys and endpoints in your source code.

Friday, 22 September 2023

Using Azure Computer Vision to perform Optical Character Recognition (OCR)

This article shows how you can use Azure Computer vision in Azure Cognitive Services to perform Optical Character Recognition (OCR). The Computer vision feature is available by adding a Computer Vision resource in Azure Portal. I have made a .NET MAUI Blazor app and the Github Repo for it is available here : https://github.com/toreaurstadboss/Ocr.Handwriting.Azure.AI.Models
Let us first look at the .csproj of the Lib project in this repo.


<Project Sdk="Microsoft.NET.Sdk.Razor">

  <PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>
  <ItemGroup>
    <SupportedPlatform Include="browser" />
  </ItemGroup>

	<ItemGroup>
		<PackageReference Include="Microsoft.Azure.CognitiveServices.Vision.ComputerVision" Version="7.0.1" />
		<PackageReference Include="Microsoft.AspNetCore.Components.Web" Version="6.0.19" />
	</ItemGroup>

</Project>


The following class generates ComputerVision clients that can be used to extract different information from streams and files containing video and images. We are going to focus on images and extracting text via OCR. Azure Computer Vision can also extract handwritten text in addition to regular text written by typewriters or text inside images and similar. Azure Computer Vision also can detect shapes in images and classify objects. This demo only focuses on text extraction form images. ComputerVisionClientFactory


using Microsoft.Azure.CognitiveServices.Vision.ComputerVision;

namespace Ocr.Handwriting.Azure.AI.Lib
{

    public interface IComputerVisionClientFactory
    {
        ComputerVisionClient CreateClient();
    }

    /// <summary>
    /// Client factory for Azure Cognitive Services - Computer vision.
    /// </summary>
    public class ComputerVisionClientFactory : IComputerVisionClientFactory
    {
        // Add your Computer Vision key and endpoint
        static string? _key = Environment.GetEnvironmentVariable("AZURE_COGNITIVE_SERVICES_VISION_KEY");
        static string? _endpoint = Environment.GetEnvironmentVariable("AZURE_COGNITIVE_SERVICES_VISION_ENDPOINT");

        public ComputerVisionClientFactory() : this(_key, _endpoint)
        {
        }

        public ComputerVisionClientFactory(string? key, string? endpoint)
        {
            _key = key;
            _endpoint = endpoint;
        }

        public ComputerVisionClient CreateClient()
        {
            if (_key == null)
            {
                throw new ArgumentNullException(_key, "The AZURE_COGNITIVE_SERVICES_VISION_KEY is not set. Set a system-level environment variable or provide this value by calling the overloaded constructor of this class.");
            }
            if (_endpoint == null)
            {
                throw new ArgumentNullException(_key, "The AZURE_COGNITIVE_SERVICES_VISION_ENDPOINT is not set. Set a system-level environment variable or provide this value by calling the overloaded constructor of this class.");
            }

            var client = Authenticate(_key!, _endpoint!);
            return client;
        }

        public static ComputerVisionClient Authenticate(string key, string endpoint) =>
            new ComputerVisionClient(new ApiKeyServiceClientCredentials(key))
            {
                Endpoint = endpoint
            };

    }
}



The setup of the endpoint and key of the Computer Vision resource is done via system-level envrionment variables. Next up, let's look at retrieving OCR text from images. Here we use ComputerVisionClient. We open up a stream of a file, an image, using File.OpenReadAsync and then the method ReadInStreamAsync of Computer vision client. The image we will load up in the app is selected by the user and the image is previewed and saved using MAUI Storage lib (inside the Appdata folder). OcrImageService.cs


using Microsoft.Azure.CognitiveServices.Vision.ComputerVision;
using Microsoft.Azure.CognitiveServices.Vision.ComputerVision.Models;
using Microsoft.Extensions.Logging;
using System.Diagnostics;
using ReadResult = Microsoft.Azure.CognitiveServices.Vision.ComputerVision.Models.ReadResult;

namespace Ocr.Handwriting.Azure.AI.Lib
{

    public interface IOcrImageService
    {
        Task<IList<ReadResult?>?> GetReadResults(string imageFilePath);
        Task<string> GetReadResultsText(string imageFilePath);
    }

    public class OcrImageService : IOcrImageService
    {
        private readonly IComputerVisionClientFactory _computerVisionClientFactory;
        private readonly ILogger<OcrImageService> _logger;

        public OcrImageService(IComputerVisionClientFactory computerVisionClientFactory, ILogger<OcrImageService> logger)
        {
            _computerVisionClientFactory = computerVisionClientFactory;
            _logger = logger;
        }

        private ComputerVisionClient CreateClient() => _computerVisionClientFactory.CreateClient();

        public async Task<string> GetReadResultsText(string imageFilePath)
        {
            var readResults = await GetReadResults(imageFilePath);
            var ocrText = ExtractText(readResults?.FirstOrDefault());
            return ocrText;
        }

        public async Task<IList<ReadResult?>?> GetReadResults(string imageFilePath)
        {
            if (string.IsNullOrWhiteSpace(imageFilePath))
            {
                return null;
            }

            try
            {
                var client = CreateClient();

                //Retrieve OCR results 

                using (FileStream stream = File.OpenRead(imageFilePath))
                {
                    var textHeaders = await client.ReadInStreamAsync(stream);
                    string operationLocation = textHeaders.OperationLocation;
                    string operationId = operationLocation[^36..]; //hat operator of C# 8.0 : this slices out the last 36 chars, which contains the guid chars which are 32 hexadecimals chars + four hyphens

                    ReadOperationResult results;

                    do
                    {
                        results = await client.GetReadResultAsync(Guid.Parse(operationId));
                        _logger.LogInformation($"Retrieving OCR results for operationId {operationId} for image {imageFilePath}");
                    }
                    while (results.Status == OperationStatusCodes.Running || results.Status == OperationStatusCodes.NotStarted);

                    IList<ReadResult?> result = results.AnalyzeResult.ReadResults;
                    return result;

                }
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.Message);
                return null;
            }
        }

        private static string ExtractText(ReadResult? readResult) => string.Join(Environment.NewLine, readResult?.Lines?.Select(l => l.Text) ?? new List<string>());

    }

}
                                           

Let's look at the MAUI Blazor project in the app. The MauiProgram.cs looks like this. MauiProgram.cs


using Ocr.Handwriting.Azure.AI.Data;
using Ocr.Handwriting.Azure.AI.Lib;
using Ocr.Handwriting.Azure.AI.Services;
using TextCopy;

namespace Ocr.Handwriting.Azure.AI;

public static class MauiProgram
{
    public static MauiApp CreateMauiApp()
    {
        var builder = MauiApp.CreateBuilder();
        builder
            .UseMauiApp<App>()
            .ConfigureFonts(fonts =>
            {
                fonts.AddFont("OpenSans-Regular.ttf", "OpenSansRegular");
            });

        builder.Services.AddMauiBlazorWebView();
#if DEBUG
        builder.Services.AddBlazorWebViewDeveloperTools();
        builder.Services.AddLogging();
#endif

        builder.Services.AddSingleton<WeatherForecastService>();
        builder.Services.AddScoped<IComputerVisionClientFactory, ComputerVisionClientFactory>();
        builder.Services.AddScoped<IOcrImageService, OcrImageService>();
        builder.Services.AddScoped<IImageSaveService, ImageSaveService>();

        builder.Services.InjectClipboard();

        return builder.Build();
    }
}



We also need some code to preview and save the image an end user chooses. The IImageService looks like this. ImageSaveService


using Microsoft.AspNetCore.Components.Forms;
using Ocr.Handwriting.Azure.AI.Models;

namespace Ocr.Handwriting.Azure.AI.Services
{

    public class ImageSaveService : IImageSaveService
    {

        public async Task<ImageSaveModel> SaveImage(IBrowserFile browserFile)
        {
            var buffers = new byte[browserFile.Size];
            var bytes = await browserFile.OpenReadStream(maxAllowedSize: 30 * 1024 * 1024).ReadAsync(buffers);
            string imageType = browserFile.ContentType;

            var basePath = FileSystem.Current.AppDataDirectory;
            var imageSaveModel = new ImageSaveModel
            {
                SavedFilePath = Path.Combine(basePath, $"{Guid.NewGuid().ToString("N")}-{browserFile.Name}"),
                PreviewImageUrl = $"data:{imageType};base64,{Convert.ToBase64String(buffers)}",
                FilePath = browserFile.Name,
                FileSize = bytes / 1024,
            };

            await File.WriteAllBytesAsync(imageSaveModel.SavedFilePath, buffers);

            return imageSaveModel;
        }

    }
}


Note the use of maxAllowedSize of IBrowserfile.OpenReadStream method, this is a good practice since IBrowserFile only supports 512 kB per default. I set it in the app to 30 MB to support some high res images too. We preview the image as base-64 here and we also save the image also. Note the use of FileSystem.Current.AppDataDirectory as base path here. Here we use nuget package Microsoft.Maui.Storage. These are the packages that is used for the MAUI Blazor project of the app. Ocr.Handwriting.Azure.AI.csproj



    <ItemGroup>
      <PackageReference Include="Microsoft.Azure.CognitiveServices.Vision.ComputerVision" Version="7.0.1" />
      <PackageReference Include="TextCopy" Version="6.2.1" />
    </ItemGroup>


The GUI looks like this : Index.razor


@page "/"
@using Ocr.Handwriting.Azure.AI.Models;
@using Microsoft.Azure.CognitiveServices.Vision.ComputerVision;
@using Microsoft.Azure.CognitiveServices.Vision.ComputerVision.Models;
@using Ocr.Handwriting.Azure.AI.Lib;
@using Ocr.Handwriting.Azure.AI.Services;
@using TextCopy;

@inject IImageSaveService ImageSaveService
@inject IOcrImageService OcrImageService 
@inject IClipboard Clipboard

<h1>Azure AI OCR Text recognition</h1>


<EditForm Model="Model" OnValidSubmit="@Submit" style="background-color:aliceblue">
    <DataAnnotationsValidator />
    <label><b>Select a picture to run OCR</b></label><br />
    <InputFile OnChange="@OnInputFile" accept=".jpeg,.jpg,.png" />
    <br />
    <code class="alert-secondary">Supported file formats: .jpeg, .jpg and .png</code>
    <br />
    @if (Model.PreviewImageUrl != null) { 
        <label class="alert-info">Preview of the selected image</label>
        <div style="overflow:auto;max-height:300px;max-width:500px">
            <img class="flagIcon" src="@Model.PreviewImageUrl" /><br />
        </div>

        <code class="alert-light">File Size (kB): @Model.FileSize</code>
        <br />
        <code class="alert-light">File saved location: @Model.SavedFilePath</code>
        <br />

        <label class="alert-info">Click the button below to start running OCR using Azure AI</label><br />
        <br />
        <button type="submit">Submit</button> <button style="margin-left:200px" type="button" class="btn-outline-info" @onclick="@CopyTextToClipboard">Copy to clipboard</button>
        <br />
        <br />
        <InputTextArea style="width:1000px;height:300px" readonly="readonly" placeholder="Detected text in the image uploaded" @bind-Value="Model!.OcrOutputText" rows="5"></InputTextArea>
    }
</EditForm>


@code {

    private IndexModel Model = new();

    private async Task OnInputFile(InputFileChangeEventArgs args)
    {       
        var imageSaveModel = await ImageSaveService.SaveImage(args.File);
        Model = new IndexModel(imageSaveModel);
        await Application.Current.MainPage.DisplayAlert($"MAUI Blazor OCR App", $"Wrote file to location : {Model.SavedFilePath} Size is: {Model.FileSize} kB", "Ok", "Cancel");
    }

    public async Task CopyTextToClipboard()
    {
        await Clipboard.SetTextAsync(Model.OcrOutputText);
        await Application.Current.MainPage.DisplayAlert($"MAUI Blazor OCR App", $"The copied text was put into the clipboard. Character length: {Model.OcrOutputText?.Length}", "Ok", "Cancel");

    }

    private async Task Submit()
    {
        if (Model.PreviewImageUrl == null || Model.SavedFilePath == null)
        {
            await Application.Current.MainPage.DisplayAlert($"MAUI Blazor OCR App", $"You must select an image first before running OCR. Supported formats are .jpeg, .jpg and .png", "Ok", "Cancel");
            return;
        }
        Model.OcrOutputText = await OcrImageService.GetReadResultsText(Model.SavedFilePath);
        StateHasChanged(); //visual refresh here
    }

}


The UI works like this. The user selects an image. As we can see by the 'accept' html attribute, the .jpeg, .jpg and .png extensions are allowed in the file input dialog. When the user selects an image, the image is saved and previewed in the UI. By hitting the Submit button, the OCR service in Azure is contacted and text is retrieved and displayed in the text area below, if any text is present in the image. A button allows copying the text into the clipboard. Here are some screenshots of the app.


Tuesday, 19 September 2023

Using Azure AI TextAnalytics and translation service to build an universal translator

This article shows code how to build a universal translator using Azure AI Cognitive Services. This includes Azure AI Textanalytics to detect languge from text input, and using Azure AI Translation services. The Github repo is here :
https://github.com/toreaurstadboss/MultiLingual.Translator
The following Nuget packages are used in the Lib project csproj file :

 <ItemGroup>
    <PackageReference Include="Azure.AI.Translation.Text" Version="1.0.0-beta.1" />
    <PackageReference Include="Microsoft.AspNetCore.Components.Web" Version="6.0.19" />
    <PackageReference Include="Azure.AI.TextAnalytics" Version="5.3.0" />
  </ItemGroup>


We are going to build a .NET 6 cross platform MAUI Blazor app. First off, we focus on the Razor Library project called 'Lib'. This project contains the library util code to detect language and translate into other language. Let us first look at creating the clients needed to detect language and to translate text. TextAnalyticsFactory.cs


using Azure;
using Azure.AI.TextAnalytics;
using Azure.AI.Translation.Text;
using System;

namespace MultiLingual.Translator.Lib
{
    public static class TextAnalyticsClientFactory
    {

        public static TextAnalyticsClient CreateClient()
        {
            string? uri = Environment.GetEnvironmentVariable("AZURE_COGNITIVE_SERVICE_ENDPOINT", EnvironmentVariableTarget.Machine);
            string? key = Environment.GetEnvironmentVariable("AZURE_COGNITIVE_SERVICE_KEY", EnvironmentVariableTarget.Machine);
            if (uri == null)
            {
                throw new ArgumentNullException(nameof(uri), "Could not get system environment variable named 'AZURE_COGNITIVE_SERVICE_ENDPOINT' Set this variable first.");
            }
            if (key == null)
            {
                throw new ArgumentNullException(nameof(uri), "Could not get system environment variable named 'AZURE_COGNITIVE_SERVICE_KEY' Set this variable first.");
            }
            var client = new TextAnalyticsClient(new Uri(uri!), new AzureKeyCredential(key!));
            return client;
        }

        public static TextTranslationClient CreateTranslateClient()
        {
            string? keyTranslate = Environment.GetEnvironmentVariable("AZURE_TRANSLATION_SERVICE_KEY", EnvironmentVariableTarget.Machine);
            string? regionForTranslationService = Environment.GetEnvironmentVariable("AZURE_TRANSLATION_SERVICE_REGION", EnvironmentVariableTarget.Machine);

            if (keyTranslate == null)
            {
                throw new ArgumentNullException(nameof(keyTranslate), "Could not get system environment variable named 'AZURE_TRANSLATION_SERVICE_KEY' Set this variable first.");
            }
            if (keyTranslate == null)
            {
                throw new ArgumentNullException(nameof(keyTranslate), "Could not get system environment variable named 'AZURE_TRANSLATION_SERVICE_REGION' Set this variable first.");
            }
            var client = new TextTranslationClient(new AzureKeyCredential(keyTranslate!), region: regionForTranslationService);
            return client;
        }

    }
}


The code assumes that there is four environment variables at the SYSTEM level of your OS. Further on, let us now look at the code to detect language. This uses a TextAnalyticsClient detect the language an input text is written in, using this client. IDetectLanguageUtil.cs


using Azure.AI.TextAnalytics;

namespace MultiLingual.Translator.Lib
{
    public interface IDetectLanguageUtil
    {
        Task<DetectedLanguage> DetectLanguage(string inputText);
        Task<double> DetectLanguageConfidenceScore(string inputText);
        Task<string> DetectLanguageIso6391(string inputText);
        Task<string> DetectLanguageName(string inputText);
    }
}


DetectLanguageUtil.cs


using Azure.AI.TextAnalytics;

namespace MultiLingual.Translator.Lib
{

    public class DetectLanguageUtil : IDetectLanguageUtil
    {

        private TextAnalyticsClient _client;

        public DetectLanguageUtil()
        {
            _client = TextAnalyticsClientFactory.CreateClient();
        }

        /// <summary>
        /// Detects language of the <paramref name="inputText"/>.
        /// </summary>
        /// <param name="inputText"></param>
        /// <remarks> <see cref="Models.LanguageCode" /> contains the language code list of languages supported</remarks>
        public async Task<DetectedLanguage> DetectLanguage(string inputText)
        {
            DetectedLanguage detectedLanguage = await _client.DetectLanguageAsync(inputText);
            return detectedLanguage;
        }

        /// <summary>
        /// Detects language of the <paramref name="inputText"/>. Returns the language name.
        /// </summary>
        /// <param name="inputText"></param>
        /// <remarks> <see cref="Models.LanguageCode" /> contains the language code list of languages supported</remarks>
        public async Task<string> DetectLanguageName(string inputText)
        {
            DetectedLanguage detectedLanguage = await DetectLanguage(inputText);
            return detectedLanguage.Name;
        }

        /// <summary>
        /// Detects language of the <paramref name="inputText"/>. Returns the language code.
        /// </summary>
        /// <param name="inputText"></param>
        /// <remarks> <see cref="Models.LanguageCode" /> contains the language code list of languages supported</remarks>
        public async Task<string> DetectLanguageIso6391(string inputText)
        {
            DetectedLanguage detectedLanguage = await DetectLanguage(inputText);
            return detectedLanguage.Iso6391Name;
        }

        /// <summary>
        /// Detects language of the <paramref name="inputText"/>. Returns the confidence score
        /// </summary>
        /// <param name="inputText"></param>
        /// <remarks> <see cref="Models.LanguageCode" /> contains the language code list of languages supported</remarks>
        public async Task<double> DetectLanguageConfidenceScore(string inputText)
        {
            DetectedLanguage detectedLanguage = await DetectLanguage(inputText);
            return detectedLanguage.ConfidenceScore;
        }

    }

}



The Iso6391 code is important when it comes to translation, which will be shown soon. But first let us look at the supported languages of Azure AI Translation services. LanguageCode.cs


namespace MultiLingual.Translator.Lib.Models
{
    /// 
    /// List of supported languages in Azure AI services
    /// https://learn.microsoft.com/en-us/azure/ai-services/translator/language-support
    /// 
    public static class LanguageCode
    {

        public const string Afrikaans = "af";
        public const string Albanian = "sq";
        public const string Amharic = "am";
        public const string Arabic = "ar";
        public const string Armenian = "hy";
        public const string Assamese = "as";
        public const string AzerbaijaniLatin = "az";
        public const string Bangla = "bn";
        public const string Bashkir = "ba";
        public const string Basque = "eu";
        public const string BosnianLatin = "bs";
        public const string Bulgarian = "bg";
        public const string CantoneseTraditional = "yue";
        public const string Catalan = "ca";
        public const string ChineseLiterary = "lzh";
        public const string ChineseSimplified = "zh-Hans";
        public const string ChineseTraditional = "zh-Hant";
        public const string chiShona = "sn";
        public const string Croatian = "hr";
        public const string Czech = "cs";
        public const string Danish = "da";
        public const string Dari = "prs";
        public const string Divehi = "dv";
        public const string Dutch = "nl";
        public const string English = "en";
        public const string Estonian = "et";
        public const string Faroese = "fo";
        public const string Fijian = "fj";
        public const string Filipino = "fil";
        public const string Finnish = "fi";
        public const string French = "fr";
        public const string FrenchCanada = "fr-ca";
        public const string Galician = "gl";
        public const string Georgian = "ka";
        public const string German = "de";
        public const string Greek = "el";
        public const string Gujarati = "gu";
        public const string HaitianCreole = "ht";
        public const string Hausa = "ha";
        public const string Hebrew = "he";
        public const string Hindi = "hi";
        public const string HmongDawLatin = "mww";
        public const string Hungarian = "hu";
        public const string Icelandic = "is";
        public const string Igbo = "ig";
        public const string Indonesian = "id";
        public const string Inuinnaqtun = "ikt";
        public const string Inuktitut = "iu";
        public const string InuktitutLatin = "iu-Latn";
        public const string Irish = "ga";
        public const string Italian = "it";
        public const string Japanese = "ja";
        public const string Kannada = "kn";
        public const string Kazakh = "kk";
        public const string Khmer = "km";
        public const string Kinyarwanda = "rw";
        /// 
        /// Fear my Bak'leth ! 
        /// 
        public const string Klingon = "tlh-Latn";
        public const string KlingonplqaD = "tlh-Piqd";
        public const string Konkani = "gom";
        public const string Korean = "ko";
        public const string KurdishCentral = "ku";
        public const string KurdishNorthern = "kmr";
        public const string KyrgyzCyrillic = "ky";
        public const string Lao = "lo";
        public const string Latvian = "lv";
        public const string Lithuanian = "lt";
        public const string Lingala = "ln";
        public const string LowerSorbian = "dsb";
        public const string Luganda = "lug";
        public const string Macedonian = "mk";
        public const string Maithili = "mai";
        public const string Malagasy = "mg";
        public const string MalayLatin = "ms";
        public const string Malayalam = "ml";
        public const string Maltese = "mt";
        public const string Maori = "mi";
        public const string Marathi = "mr";
        public const string MongolianCyrillic = "mn-Cyrl";
        public const string MongolianTraditional = "mn-Mong";
        public const string Myanmar = "my";
        public const string Nepali = "ne";
        public const string Norwegian = "nb";
        public const string Nyanja = "nya";
        public const string Odia = "or";
        public const string Pashto = "ps";
        public const string Persian = "fa";
        public const string Polish = "pl";
        public const string PortugueseBrazil = "pt";
        public const string PortuguesePortugal = "pt-pt";
        public const string Punjabi = "pa";
        public const string QueretaroOtomi = "otq";
        public const string Romanian = "ro";
        public const string Rundi = "run";
        public const string Russian = "ru";
        public const string SamoanLatin = "sm";
        public const string SerbianCyrillic = "sr-Cyrl";
        public const string SerbianLatin = "sr-Latn";
        public const string Sesotho = "st";
        public const string SesothosaLeboa = "nso";
        public const string Setswana = "tn";
        public const string Sindhi = "sd";
        public const string Sinhala = "si";
        public const string Slovak = "sk";
        public const string Slovenian = "sl";
        public const string SomaliArabic = "so";
        public const string Spanish = "es";
        public const string SwahiliLatin = "sw";
        public const string Swedish = "sv";
        public const string Tahitian = "ty";
        public const string Tamil = "ta";
        public const string TatarLatin = "tt";
        public const string Telugu = "te";
        public const string Thai = "th";
        public const string Tibetan = "bo";
        public const string Tigrinya = "ti";
        public const string Tongan = "to";
        public const string Turkish = "tr";
        public const string TurkmenLatin = "tk";
        public const string Ukrainian = "uk";
        public const string UpperSorbian = "hsb";
        public const string Urdu = "ur";
        public const string UyghurArabic = "ug";
        public const string UzbekLatin = "uz";
        public const string Vietnamese = "vi";
        public const string Welsh = "cy";
        public const string Xhosa = "xh";
        public const string Yoruba = "yo";
        public const string YucatecMaya = "yua";
        public const string Zulu = "zu";
    }
}


As there are about 5-10 000 languages in the World, the list above shows that Azure AI translation services supports about 130 of these, which is 1-2 % of the total amount of languages. Of course, the languages supported by Azure AI are also including the most spoken languages in the World. Let us look at the translation util code next. ITranslateUtil.cs


namespace MultiLingual.Translator.Lib
{
    public interface ITranslateUtil
    {
        Task<string?> Translate(string targetLanguage, string inputText, string? sourceLanguage = null);
    }
}


TranslateUtil.cs


using Azure.AI.Translation.Text;
using MultiLingual.Translator.Lib.Models;

namespace MultiLingual.Translator.Lib
{

    public class TranslateUtil : ITranslateUtil
    {
        private TextTranslationClient _client;


        public TranslateUtil()
        {
            _client = TextAnalyticsClientFactory.CreateTranslateClient();
        }

        /// <summary>
        /// Translates text using Azure AI Translate services. 
        /// </summary>
        /// <param name="targetLanguage"><see cref="LanguageCode" for a list of supported languages/></param>
        /// <param name="inputText"></param>
        /// <param name="sourceLanguage">Pass in null here to auto detect the source language</param>
        /// <returns></returns>
        public async Task<string?> Translate(string targetLanguage, string inputText, string? sourceLanguage = null)
        {
            var translationOfText = await _client.TranslateAsync(targetLanguage, inputText, sourceLanguage);
            if (translationOfText?.Value == null)
            {
                return null;
            }
            var translation = translationOfText.Value.SelectMany(l => l.Translations).Select(l => l.Text)?.ToList();
            string? translationText = translation?.FlattenString();
            return translationText;
        }

    }
}


We use a little helper extension method here too : StringExtensions.cs


using System.Text;

namespace MultiLingual.Translator.Lib
{
    public static class StringExtensions
    {

        /// <summary>
        /// Merges a collection of lines into a flattened string separating each line by a specified line separator.
        /// Newline is deafult
        /// </summary>
        /// <param name="inputLines"></param>
        /// <param name="lineSeparator"></param>
        /// <returns></returns>
        public static string? FlattenString(this IEnumerable<string>? inputLines, string lineSeparator = "\n")
        {
            if (inputLines == null || !inputLines.Any())
            {
                return null;
            }
            var flattenedString = inputLines?.Aggregate(new StringBuilder(),
                (sb, l) => sb.AppendLine(l + lineSeparator),
                sb => sb.ToString().Trim());

            return flattenedString;
        }

    }
}


Here are some tests for detecting language : DetectLanguageUtilTests.cs

  
using Azure.AI.TextAnalytics;
using FluentAssertions;

namespace MultiLingual.Translator.Lib.Test
{
    public class DetectLanguageUtilTests
    {

        private DetectLanguageUtil _detectLanguageUtil;

        public DetectLanguageUtilTests()
        {
            _detectLanguageUtil = new DetectLanguageUtil();
        }

        [Theory]
        [InlineData("Donde esta la playa", "es", "Spanish")]
        [InlineData("Jeg er fra Trøndelag og jeg liker brunost", "no", "Norwegian")]
        public async Task DetectLanguageDetailsSucceeds(string text, string expectedLanguageIso6391, string expectedLanguageName)
        {
            string? detectedLangIso6391 = await _detectLanguageUtil.DetectLanguageIso6391(text);
            detectedLangIso6391.Should().Be(expectedLanguageIso6391);
            string? detectedLangName = await _detectLanguageUtil.DetectLanguageName(text);
            detectedLangName.Should().Be(expectedLanguageName);
        }

        [Theory]
        [InlineData("Du hast mich", "de", "German")]
        public async Task DetectLanguageSucceeds(string text, string expectedLanguageIso6391, string expectedLanguageName)
        {
            DetectedLanguage detectedLanguage = await _detectLanguageUtil.DetectLanguage(text);
            detectedLanguage.Iso6391Name.Should().Be(expectedLanguageIso6391);            
            detectedLanguage.Name.Should().Be(expectedLanguageName);
        }

    }
}  
  

And here are some translation util tests : TranslateUtilTests.cs


using FluentAssertions;
using MultiLingual.Translator.Lib.Models;

namespace MultiLingual.Translator.Lib.Test
{

    public class TranslateUtilTests
    {

        private TranslateUtil _translateUtil;

        public TranslateUtilTests()
        {
            _translateUtil = new TranslateUtil();                
        }

        [Theory]
        [InlineData("Jeg er fra Norge og jeg liker brunost", "i'm from norway and i like brown cheese", LanguageCode.Norwegian,  LanguageCode.English)]
        [InlineData("Jeg er fra Norge og jeg liker brunost", "i'm from norway and i like brown cheese", null, LanguageCode.English)] //auto detect language is tested here
        [InlineData("Ich bin aus Hamburg und ich liebe bier", "i'm from hamburg and i love beer", LanguageCode.German, LanguageCode.English)]
        [InlineData("Ich bin aus Hamburg und ich liebe bier", "i'm from hamburg and i love beer", null, LanguageCode.English)] //Auto detect source language is tested here
        [InlineData("tlhIngan maH", "we are klingons", LanguageCode.Klingon, LanguageCode.English)] //Klingon force !
        public async Task TranslationReturnsExpected(string input, string expectedTranslation, string sourceLanguage, string targetLanguage)
        {
            string? translation = await _translateUtil.Translate(targetLanguage, input, sourceLanguage);
            translation.Should().NotBeNull();
            translation.Should().BeEquivalentTo(expectedTranslation);
        }

    }
}
  

Over to the UI. The app is made with MAUI Blazor. Here are some models for the app : LanguageInputModel.cs


namespace MultiLingual.Translator.Models
{
    public class LanguageInputModel
    {
        public string InputText { get; set; }

        public string DetectedLanguageInfo { get; set; }

        public string DetectedLanguageIso6391 { get; set; }

        public string TargetLanguage { get; set; }

        public string TranslatedText { get; set; }

    }
}



NameValue.cs


namespace MultiLingual.Translator.Models
{
    public class NameValue
    {
        public string Name { get; set; }
        public string Value { get; set; }
    }
}


The UI consists of this razor code in, written for Blazor MAUI app. Index.razor


@page "/"
@inject ITranslateUtil TransUtil
@inject IDetectLanguageUtil DetectLangUtil
@inject IJSRuntime JS

@using MultiLingual.Translator.Lib;
@using MultiLingual.Translator.Lib.Models;
@using MultiLingual.Translator.Models;

<h1>Azure AI Text Translation</h1>

<EditForm Model="@Model" OnValidSubmit="@Submit" class="form-group" style="background-color:aliceblue;">
    <DataAnnotationsValidator />
    <ValidationSummary />

    <div class="form-group row">
        <label for="Model.InputText">Text to translate</label>
        <InputTextArea @bind-Value="Model!.InputText" placeholder="Enter text to translate" @ref="inputTextRef" id="textToTranslate" rows="5" />
    </div>

    <div class="form-group row">
        <span>Detected language of text to translate</span>
        <InputText class="languageLabelText" readonly="readonly" placeholder="The detected language of the text to translate" @bind-Value="Model!.DetectedLanguageInfo"></InputText>
        @if (Model.DetectedLanguageInfo != null){
            <img src="@FlagIcon" class="flagIcon" />
        }
    </div>
    <br />
    
    <div class="form-group row">
        <span>Translate into language</span>
        <InputSelect placeholder="Choose the target language"  @bind-Value="Model!.TargetLanguage">
            @foreach (var item in LanguageCodes){
                <option value="@item.Value">@item.Name</option>
            }
        </InputSelect>
        <br />
          @if (Model.TargetLanguage != null){
            <img src="@TargetFlagIcon" class="flagIcon" />
        }
    </div>
    <br />

    <div class="form-group row">
        <span>Translation</span>
        <InputTextArea readonly="readonly" placeholder="The translated text target language" @bind-Value="Model!.TranslatedText" rows="5"></InputTextArea>
    </div>

    <button type="submit" class="submitButton">Submit</button>

</EditForm>

@code {
    private Azure.AI.TextAnalytics.TextAnalyticsClient _client;

    private InputTextArea inputTextRef;

    public LanguageInputModel Model { get; set; } = new();

    private string FlagIcon {
        get
        {
            return $"images/flags/png100px/{Model.DetectedLanguageIso6391}.png";
        }
    }

    private string TargetFlagIcon {
        get
        {
            return $"images/flags/png100px/{Model.TargetLanguage}.png";
        }
    }

    private List<NameValue> LanguageCodes = typeof(LanguageCode).GetFields().Select(f => new NameValue {
	 Name = f.Name,
	 Value = f.GetValue(f)?.ToString(),
	}).OrderBy(f => f.Name).ToList();


    private async void Submit()
    {
        var detectedLanguage = await DetectLangUtil.DetectLanguage(Model.InputText);
        Model.DetectedLanguageInfo = $"{detectedLanguage.Iso6391Name} {detectedLanguage.Name}";
        Model.DetectedLanguageIso6391 = detectedLanguage.Iso6391Name;
        if (_client == null)
        {
            _client = TextAnalyticsClientFactory.CreateClient();
        }
        Model.TranslatedText = await TransUtil.Translate(Model.TargetLanguage, Model.InputText, detectedLanguage.Iso6391Name);

        StateHasChanged();
    }

    protected override async Task OnAfterRenderAsync(bool firstRender)
    {
        if (firstRender)
        {
            Model.TargetLanguage = LanguageCode.English;
            await JS.InvokeVoidAsync("exampleJsFunctions.focusElement", inputTextRef?.AdditionalAttributes.FirstOrDefault(a => a.Key?.ToLower() == "id").Value);
            StateHasChanged();
        }
    }

}


Finally, a screenshot how the app looks like : You enter the text to translate, then the detected language is shown after you hit Submit. You can select the target language to translate the text into. English is selected as default. The Iso6391 code of the selected language is shown as a flag icon, if there exists a 1:1 mapping between the Iso6391 code and the flag icons available in the app. The top flag show the detected language via the Iso6391 code, IF there is a 1:1 mapping between this code and the available Flag icons.