Pii - Detecting Personally Identifiable Information using Azure Cognitive Services

This article will look at detecting Person Identifiable Information (Pii) using Azure Cognitive Services. I have created a demo using .NET Maui Blazor has been created and the Github repo is here:
https://github.com/toreaurstadboss/PiiDetectionDemo

Person Identifiable Information (Pii) is desired to detect and also redact, that is using censorship or obscuring Pii to prepare documents for publication. The Pii feature in Azure Cognitive Services is a part of the Language resource service. A quickstart for using Pii is available here:
https://learn.microsoft.com/en-us/azure/ai-services/language-service/personally-identifiable-information/quickstart?pivots=programming-language-csharp

After creating the Language resource, look up the keys and endpoints for you service. Using Azure CLI inside Cloud shell, you can enter this command to find the keys, in Azure many services has got two keys you can exchange with new keys through regeneration:

az cognitiveservices account keys list --resource-group SomeAzureResourceGroup --name SomeAccountAzureCognitiveServices
This is how you can query after endpoint of language resource using Azure CLI : az cognitiveservices account show --query "properties.endpoint" --resource-group SomeAzureResourceGroup --name SomeAccountAzureCognitiveServices
Next, the demo of this article. Connecting to the Pii Removal Text Analytics is possible using this Nuget package (REST calls can also be done manually): - Azure.AI.TextAnalytics version 5.3.0 Here is the other Nugets of my Demo included from the .csproj file :

PiiDetectionDemo.csproj

  <ItemGroup>
        <PackageReference Include="Azure.AI.TextAnalytics" Version="5.3.0" />
        <PackageReference Include="Microsoft.Maui.Controls" Version="$(MauiVersion)" />
        <PackageReference Include="Microsoft.Maui.Controls.Compatibility" Version="$(MauiVersion)" />
        <PackageReference Include="Microsoft.AspNetCore.Components.WebView.Maui" Version="$(MauiVersion)" />
        <PackageReference Include="Microsoft.Extensions.Logging.Debug" Version="8.0.0" />
    </ItemGroup>

A service using this Pii removal feature is simply making use of a TextAnalyticsClient and method RecognizePiiEntitiesAsync.

PiiRemovalTextClientService.cs IPiiRemovalTextClientService.cs

using Azure;
using Azure.AI.TextAnalytics;

namespace PiiDetectionDemo.Util
{
    public interface IPiiRemovalTextAnalyticsClientService
    {
        Task<Response<PiiEntityCollection>> RecognizePiiEntitiesAsync(string? document, string? language);
    }
}


namespace PiiDetectionDemo.Util
{
    public class PiiRemovalTextAnalyticsClientService : IPiiRemovalTextAnalyticsClientService
    {

        private TextAnalyticsClient _client;

        public PiiRemovalTextAnalyticsClientService()
        {
            var azureEndpoint = Environment.GetEnvironmentVariable("AZURE_COGNITIVE_SERVICE_ENDPOINT");
            var azureKey = Environment.GetEnvironmentVariable("AZURE_COGNITIVE_SERVICE_KEY");

            if (string.IsNullOrWhiteSpace(azureEndpoint))
            {
                throw new ArgumentNullException(nameof(azureEndpoint), "Missing system environment variable: AZURE_COGNITIVE_SERVICE_ENDPOINT");
            }
            if (string.IsNullOrWhiteSpace(azureKey))
            {
                throw new ArgumentNullException(nameof(azureKey), "Missing system environment variable: AZURE_COGNITIVE_SERVICE_KEY");
            }

            _client = new TextAnalyticsClient(new Uri(azureEndpoint), new AzureKeyCredential(azureKey));
        }

        public async Task<Response<PiiEntityCollection>> RecognizePiiEntitiesAsync(string? document, string? language)
        {
            var piiEntities = await _client.RecognizePiiEntitiesAsync(document, language);
            return piiEntities;
        }

    }
}

The UI codebehind of the razor component page showing the UI looks like this:

Home.razor.cs

using Azure;
using Microsoft.AspNetCore.Components;
using PiiDetectionDemo.Models;
using PiiDetectionDemo.Util;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace PiiDetectionDemo.Components.Pages
{
    public partial class Home
    {

        private IndexModel Model = new();
        private bool isProcessing = false;
        private bool isSearchPerformed = false;

        private async Task Submit()
        {
            isSearchPerformed = false;
            isProcessing = true;
            try
            {
                var response = await _piiRemovalTextAnalyticsClientService.RecognizePiiEntitiesAsync(Model.InputText, null);
                Model.RedactedText = response?.Value?.RedactedText;
                Model.UpdateHtmlRedactedText();
                Model.AnalysisResult = response?.Value;
                StateHasChanged();
            }
            catch (Exception ex)
            {
                await Console.Out.WriteLineAsync(ex.ToString());
            }
            isProcessing = false;
            isSearchPerformed = true;
        }

        private void removeWhitespace(ChangeEventArgs args)
        {
            Model.InputText = args.Value?.ToString()?.CleanupAllWhiteSpace();
            StateHasChanged();
        }



    }
}

To get the redacted or censored text void of any Pii that the Pii detection feature was able to detect, access the Value of type Azure.AI.TextAnalytics.PiiEntityCollection. Inside this object, the string RedactedText contains the censored / redacted text. The IndexModel looks like this :

using Azure.AI.TextAnalytics;
using Microsoft.AspNetCore.Components;
using PiiDetectionDemo.Util;
using System.ComponentModel.DataAnnotations;
using System.Text;

namespace PiiDetectionDemo.Models
{

    public class IndexModel
    {

        [Required]
        public string? InputText { get; set; }

        public string? RedactedText { get; set; }

        public string? HtmlRedactedText { get; set; }

        public MarkupString HtmlRedactedTextMarkupString { get; set; }

        public void UpdateHtmlRedactedText()
        {
            var sb = new StringBuilder(RedactedText);
            if (AnalysisResult != null && RedactedText != null)
            {
                foreach (var piiEntity in AnalysisResult.OrderByDescending(a => a.Offset))
                {
                    sb.Insert(piiEntity.Offset + piiEntity.Length, "</b></span>");
                    sb.Insert(piiEntity.Offset, $"<span style='background-color:lightgray;border:1px solid black;corner-radius:2px; color:{GetBackgroundColor(piiEntity)}' title='{piiEntity.Category}: {piiEntity.SubCategory} Confidence: {piiEntity.ConfidenceScore} Redacted Text: {piiEntity.Text}'><b>");
                }
            }
            HtmlRedactedText = sb.ToString()?.CleanupAllWhiteSpace();    
            HtmlRedactedTextMarkupString = new MarkupString(HtmlRedactedText ?? string.Empty);
        }

        private string GetBackgroundColor(PiiEntity piiEntity)
        {
            if (piiEntity.Category == PiiEntityCategory.PhoneNumber)
            {
                return "yellow";
            }
            if (piiEntity.Category == PiiEntityCategory.Organization)
            {
                return "orange";
            }
            if (piiEntity.Category == PiiEntityCategory.Address)
            {
                return "green";
            }
            return "gray";                   
        }

        public long ExecutionTime { get; set; }
        public PiiEntityCollection? AnalysisResult { get; set; }

    }
}

Frontend UI looks like this: Home.razor

@page "/"
@using PiiDetectionDemo.Util

@inject IPiiRemovalTextAnalyticsClientService _piiRemovalTextAnalyticsClientService;

<h3>Azure HealthCare Text Analysis - Pii detection feature - Azure Cognitive Services</h3>

<em>Pii = Person identifiable information</em>

<EditForm Model="@Model" OnValidSubmit="@Submit">
    <DataAnnotationsValidator />
    <ValidationSummary />

    <div class="form-group row">
        <label><strong>Text input</strong></label>
        <InputTextArea @oninput="removeWhitespace" class="overflow-scroll" style="max-height:500px;max-width:900px;font-size: 10pt;font-family:Verdana, Geneva, Tahoma, sans-serif" @bind-Value="@Model.InputText" rows="5" />
    </div>

    <div class="form-group row">
        <div class="col">
            <br />
            <button class="btn btn-outline-primary" type="submit">Run</button>
        </div>
        <div class="col">
        </div>
        <div class="col">
        </div>
    </div>

    <br />

    @if (isProcessing)
    {

        <div class="progress" style="max-width: 90%">
            <div class="progress-bar progress-bar-striped progress-bar-animated"
                 style="width: 100%; background-color: green">
                Retrieving result from Azure Text Analysis Pii detection feature. Processing..
            </div>
        </div>
        <br />

    }

    <div class="form-group row">
        <label><strong>Analysis result</strong></label>

        @if (isSearchPerformed)
        {
            <br />
            <b>Execution time took: @Model.ExecutionTime ms (milliseconds)</b>

            <br />
            <br />

            <b>Redacted text (Pii removed)</b>
            <br />

            <div class="form-group row">
               <label><strong>Categorized Pii redacted text</strong></label>
               <div>
               @Model.HtmlRedactedTextMarkupString
               </div>
            </div>

            <br />
            <br />

            <table class="table table-striped table-dark table-hover">
                <thead>
                <th>Pii text</th>
                <th>Category</th>
                <th>SubCategory</th>
                <th>Offset</th>
                <th>Length</th>
                <th>ConfidenceScore</th>
                </thead>
                <tbody>
                    @if (Model.AnalysisResult != null) {
                        @foreach (var entity in Model.AnalysisResult)
                        {
                            <tr>
                                <td>@entity.Text</td>
                                <td>@entity.Category.ToString()</td>
                                <td>@entity.SubCategory</td>
                                <td>@entity.Offset</td>
                                <td>@entity.Length</td>
                                <td>@entity.ConfidenceScore</td>                                        
                            </tr>
                        }
                    }
                </tbody>
            </table>

        }
    </div>

</EditForm>

The Demo uses Bootstrap 5 to build up a HTML table styled and showing the Azure.AI.TextAnalytics.PiiEntity properties.

Implementing Basic Auth in Core WCF

WCF or Windows Communication Foundation was released initially in 2006 and was an important part of .NET Framework to create serverside services. It supports a lot of different protocols, not only HTTP(S), but also Net.Tcp, Msmq, Named pipes and more.

Sadly, .NET Core 1, when released in 2016, did not include WCF. The use of WCF has been more and more replaced by REST API over HTTP(S) using JWT tokens and not SAML.

But a community driven project supported by a multitude of companies including Microsoft and Amazon Web Services has been working on the Core WCF project and this project is starting to gain some more use, also allowing companies to migrate their platform services over to .NET.

I have looked at some basic stuff though, namely Basic Auth in Core WCF, and actually there is no working code sample for this. I have tapped into the ASP.NET Core pipeline to make it work by studying different code samples which has made part of it work, and I got it working. In this article I will explain how.

I use GenericIdentity to make it work. On the client side I have this extension method where I pass the username and password inside the soap envelope. I use .net6 client and service and service use CoreWCF version 1.5.1.

Source code for demo client is here: https://github.com/toreaurstadboss/CoreWCFWebClient1

The client is an ASP.NET Core MVC client who has added a Core WCF service as a connected service, generating a ServiceClient. The same type of service reference seen in .NET Framework in other words.

Client side setup for Core WCF Basic Auth

Source code for demo service is here: https://github.com/toreaurstadboss/CoreWCFService1

Extension method WithBasicAuth:
BasicHttpBindingClientFactory.cs

using System.ServiceModel;
using System.ServiceModel.Channels;

namespace CoreWCFWebClient1.Extensions
{
    public static class BasicHttpBindingClientFactory
    {

        /// <summary>
        /// Creates a basic auth client with credentials set in header Authorization formatted as 'Basic [base64encoded username:password]'
        /// Makes it easier to perform basic auth in Asp.NET Core for WCF
        /// </summary>
        /// <param name="username"></param>
        /// <param name="password"></param>
        /// <returns></returns>
        public static TServiceImplementation WithBasicAuth<TServiceContract, TServiceImplementation>(this TServiceImplementation client, string username, string password)
              where TServiceContract : class
                where TServiceImplementation : ClientBase<TServiceContract>, new()
        {
            string clientUrl = client.Endpoint.Address.Uri.ToString();

            var binding = new BasicHttpsBinding();
            binding.Security.Mode = BasicHttpsSecurityMode.Transport;
            binding.Security.Transport.ClientCredentialType = HttpClientCredentialType.Basic;

            string basicHeaderValue = "Basic " + Base64Encode($"{username}:{password}");
            var eab = new EndpointAddressBuilder(new EndpointAddress(clientUrl));
            eab.Headers.Add(AddressHeader.CreateAddressHeader("Authorization",  // Header Name
                string.Empty,           // Namespace
                basicHeaderValue));  // Header Value
            var endpointAddress = eab.ToEndpointAddress();

            var clientWithConfiguredBasicAuth = (TServiceImplementation) Activator.CreateInstance(typeof(TServiceImplementation), binding, endpointAddress)!;
            clientWithConfiguredBasicAuth.ClientCredentials.UserName.UserName = username;
            clientWithConfiguredBasicAuth.ClientCredentials.UserName.Password = username;

            return clientWithConfiguredBasicAuth;
        }

        private static string Base64Encode(string plainText)
        {
            var plainTextBytes = System.Text.Encoding.UTF8.GetBytes(plainText);
            return Convert.ToBase64String(plainTextBytes);
        }

    }
}

Example call inside a razor file in a .net6 web client, I made client and service from the WCF template :
Index.cshtml

@{

    string username = "someuser";
    string password = "somepassw0rd";

    var client = new ServiceClient().WithBasicAuth<IService, ServiceClient>(username, password);

    var result = await client.GetDataAsync(42);

    <h5>@Html.Raw(result)</h5>
}

I manage to set the identity via the call above, here is a screenshot showing this :

Setting up Basic Auth for serverside

Let's look at the serverside, it was created to start with as an ASP.NET Core .NET 6 with MVC Views solution. I added these Nugets to add CoreWCF, showing the entire .csproj since it also includes some important using statements :
CoreWCFService1.csproj

<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>true</ImplicitUsings>
  </PropertyGroup>
  <ItemGroup>
    <Using Include="CoreWCF" />
    <Using Include="CoreWCF.Configuration" />
    <Using Include="CoreWCF.Channels" />
    <Using Include="CoreWCF.Description" />
    <Using Include="System.Runtime.Serialization " />
    <Using Include="CoreWCFService1" />
    <Using Include="Microsoft.Extensions.DependencyInjection.Extensions" />
  </ItemGroup>
  <ItemGroup>
    <PackageReference Include="CoreWCF.Primitives" Version="1.5.1" />
    <PackageReference Include="CoreWCF.Http" Version="1.5.1" />
  </ItemGroup>
</Project>

Next up, in the file Program.cs different setup is added to add Basic Auth. In Program.cs , basic auth is set up in these code lines :
Program.cs

builder.Services.AddSingleton<IUserRepository, UserRepository>();

builder.Services.AddAuthentication("Basic").
            AddScheme<AuthenticationSchemeOptions, BasicAuthenticationHandler>
            ("Basic", null);
             

This adds authentication in services. We also make sure to add authentication itself after WebApplicationBuilder has been built, making sure also to set AllowSynchronousIO to true as usual. Below is listet the pipline setup of authentication, the StartsWithSegments should of course be adjusted in case you have multiple services:
Program.cs

app.Use(async (context, next) =>
{
    // Only check for basic auth when path is for the TransportWithMessageCredential endpoint only
    if (context.Request.Path.StartsWithSegments("/Service.svc"))
    {
        // Check if currently authenticated
        var authResult = await context.AuthenticateAsync("Basic");
        if (authResult.None)
        {
            // If the client hasn't authenticated, send a challenge to the client and complete request
            await context.ChallengeAsync("Basic");
            return;
        }
    }
    // Call the next delegate/middleware in the pipeline.
    // Either the request was authenticated of it's for a path which doesn't require basic auth
    await next(context);
});

We set up the servicemodel security like this to support transport mode security with the basic client credentials type.
Program.cs

 
app.UseServiceModel(serviceBuilder =>
{
    var basicHttpBinding = new BasicHttpBinding();
    basicHttpBinding.Security.Mode = BasicHttpSecurityMode.Transport;
    basicHttpBinding.Security.Transport.ClientCredentialType = HttpClientCredentialType.Basic;
    serviceBuilder.AddService<Service>(options =>
    {
        options.DebugBehavior.IncludeExceptionDetailInFaults = true;
    });
    serviceBuilder.AddServiceEndpoint<Service, IService>(basicHttpBinding, "/Service.svc");

    var serviceMetadataBehavior = app.Services.GetRequiredService<ServiceMetadataBehavior>();
    serviceMetadataBehavior.HttpsGetEnabled = true;
});
  

The BasicAuthenticationHandler looks like this:
BasicAuthenticationHandler.cs

	
using Microsoft.AspNetCore.Authentication;
using Microsoft.Extensions.Options;
using System.Security.Claims;
using System.Security.Principal;
using System.Text;
using System.Text.Encodings.Web;

public class BasicAuthenticationHandler : AuthenticationHandler<AuthenticationSchemeOptions>
{

    private readonly IUserRepository _userRepository;
    public BasicAuthenticationHandler(IOptionsMonitor<AuthenticationSchemeOptions> options,
        ILoggerFactory logger,
        UrlEncoder encoder,
        ISystemClock clock, IUserRepository userRepository) :
       base(options, logger, encoder, clock)
    {
        _userRepository = userRepository;
    }

    protected async override Task<AuthenticateResult> HandleAuthenticateAsync()
    {
        string? authTicketFromSoapEnvelope = await Request!.GetAuthenticationHeaderFromSoapEnvelope();

        if (authTicketFromSoapEnvelope != null && authTicketFromSoapEnvelope.StartsWith("basic", StringComparison.OrdinalIgnoreCase))
        {
            var token = authTicketFromSoapEnvelope.Substring("Basic ".Length).Trim();
            var credentialsAsEncodedString = Encoding.UTF8.GetString(Convert.FromBase64String(token));
            var credentials = credentialsAsEncodedString.Split(':');
            if (await _userRepository.Authenticate(credentials[0], credentials[1]))
            {
                var identity = new GenericIdentity(credentials[0]);
                var claimsPrincipal = new ClaimsPrincipal(identity);
                var ticket = new AuthenticationTicket(claimsPrincipal, Scheme.Name);
                return await Task.FromResult(AuthenticateResult.Success(ticket));
            }
        }

        return await Task.FromResult(AuthenticateResult.Fail("Invalid Authorization Header"));
    }

    protected override Task HandleChallengeAsync(AuthenticationProperties properties)
    {
        Response.StatusCode = 401;
        Response.Headers.Add("WWW-Authenticate", "Basic realm=\"thoushaltnotpass.com\"");
        Context.Response.WriteAsync("You are not logged in via Basic auth").Wait();
        return Task.CompletedTask;
    }

}

This authentication handler has got a flaw, if you enter the wrong password and username you get a 500 internal server error instead of the 401. I have not found out how to fix this yet.. Authenticate.Fail seems to short-circuit everything in case you enter wrong credentials. The _userRepository.Authenticate method is implemented as a dummy implementation, the user repo could for example do a database connection to look up the user via the provided credentials or some other means, maybe via ASP.NET Core MemberShipProvider ? The user repo looks like this:
(I)UserRepository.cs

  public interface IUserRepository
    {

        public Task<bool> Authenticate(string username, string password);
    }

    public class UserRepository : IUserRepository
    {
        public Task<bool> Authenticate(string username, string password)
        {
            //TODO: some dummie auth mechanism used here, make something more realistic such as DB user repo lookup or similar
            if (username == "someuser" && password == "somepassw0rd")
            {
                return Task.FromResult(true);
            }
            return Task.FromResult(false);
        }
    }
    
    
 

So I have implemented basic auth via reading out the credentials via Auth header inside soap envelope. I circumvent a lot of the Core WCF Auth by perhaps relying too much on the ASP.Net Core pipeline instead. Remember, WCF has to interop some with the ASP.NET Core pipeline to make it work properly and as long as we satisfy the demands of both the WCF and ASP.NET Core pipelines, we can make the authentication work. I managed to set the username via setting claims in the expected places of ServiceSecurityContext and CurrentPrincipal. The WCF service looks like this, note the use of the [Autorize] attribute :
Service.cs

   
public class Service : IService
 {

     [Authorize]
     public string GetData(int value)
     {
         return $"You entered: {value}. <br />The client logged in with transport security with BasicAuth with https (BasicHttpsBinding).<br /><br />The username is set inside ServiceSecurityContext.Current.PrimaryIdentity.Name: {ServiceSecurityContext.Current.PrimaryIdentity.Name}. <br /> This username is also stored inside Thread.CurrentPrincipal.Identity.Name: {Thread.CurrentPrincipal?.Identity?.Name}";
     }

     public CompositeType GetDataUsingDataContract(CompositeType composite)
     {
         if (composite == null)
         {
             throw new ArgumentNullException("composite");
         }
         if (composite.BoolValue)
         {
             composite.StringValue += "Suffix";
         }
         return composite;
     }
 }
   

I am mainly satisfied with this setup as it though is not optimal since ASP.NET Core don't seem to be able to work together with CoreWCF properly, instead we add the authentication as a soap envelope authorization header which we read out. I used some time to read out the authentication header, this is done on the serverside with the following extension method :
HttpRequestExtensions.cs

 
 
using System.IO.Pipelines;
using System.Text;
using System.Xml.Linq;

public static class HttpRequestExtensions
{

    public static async Task<string?> GetAuthenticationHeaderFromSoapEnvelope(this HttpRequest request)
    {
        ReadResult requestBodyInBytes = await request.BodyReader.ReadAsync();
        string body = Encoding.UTF8.GetString(requestBodyInBytes.Buffer.FirstSpan);
        request.BodyReader.AdvanceTo(requestBodyInBytes.Buffer.Start, requestBodyInBytes.Buffer.End);

        string authTicketFromHeader = null;

        if (body?.Contains(@"http://schemas.xmlsoap.org/soap/envelope/") == true)
        {
            XNamespace ns = "http://schemas.xmlsoap.org/soap/envelope/";
            var soapEnvelope = XDocument.Parse(body);
            var headers = soapEnvelope.Descendants(ns + "Header").ToList();

            foreach (var header in headers)
            {
                var authorizationElement = header.Element("Authorization");
                if (!string.IsNullOrWhiteSpace(authorizationElement?.Value))
                {
                    authTicketFromHeader = authorizationElement.Value;
                    break;
                }
            }
        }

        return authTicketFromHeader;
    }

} 
 
 

Note the use of BodyReader and method AdvanceTo. This was the only way to rewind the Request stream after reading the HTTP soap envelope header for Authorization, it took me hours to figure out why this failed in ASP.NET Core pipeline, until I found some tips in a Github discussion thread on Core WCF mentioning the error and a suggestion in a comment there. See more documentation about BodyWriter and BodyReader here from MVP Steve Gordon here: https://www.stevejgordon.co.uk/using-the-bodyreader-and-bodywriter-in-asp-net-core-3-0