https://github.com/toreaurstadboss/PiiDetectionDemo
Person Identifiable Information (Pii) is desired to detect and also redact, that is using censorship or obscuring Pii to prepare documents for publication. The Pii feature in Azure Cognitive Services is a part of the Language resource service. A quickstart for using Pii is available here:
https://learn.microsoft.com/en-us/azure/ai-services/language-service/personally-identifiable-information/quickstart?pivots=programming-language-csharp
After creating the Language resource, look up the keys and endpoints for you service. Using Azure CLI inside Cloud shell, you can enter this command to find the keys, in Azure many services has got two keys you can exchange with new keys through regeneration:
az cognitiveservices account keys list --resource-group SomeAzureResourceGroup --name SomeAccountAzureCognitiveServices
This is how you can query after endpoint of language resource using Azure CLI :
az cognitiveservices account show --query "properties.endpoint" --resource-group SomeAzureResourceGroup --name SomeAccountAzureCognitiveServices
Next, the demo of this article. Connecting to the Pii Removal Text Analytics is possible using this Nuget package (REST calls can also be done manually): - Azure.AI.TextAnalytics version 5.3.0 Here is the other Nugets of my Demo included from the .csproj file :
PiiDetectionDemo.csproj
<ItemGroup>
<PackageReference Include="Azure.AI.TextAnalytics" Version="5.3.0" />
<PackageReference Include="Microsoft.Maui.Controls" Version="$(MauiVersion)" />
<PackageReference Include="Microsoft.Maui.Controls.Compatibility" Version="$(MauiVersion)" />
<PackageReference Include="Microsoft.AspNetCore.Components.WebView.Maui" Version="$(MauiVersion)" />
<PackageReference Include="Microsoft.Extensions.Logging.Debug" Version="8.0.0" />
</ItemGroup>
A service using this Pii removal feature is simply making use of a TextAnalyticsClient and method RecognizePiiEntitiesAsync.
PiiRemovalTextClientService.cs IPiiRemovalTextClientService.cs
using Azure;
using Azure.AI.TextAnalytics;
namespace PiiDetectionDemo.Util
{
public interface IPiiRemovalTextAnalyticsClientService
{
Task<Response<PiiEntityCollection>> RecognizePiiEntitiesAsync(string? document, string? language);
}
}
namespace PiiDetectionDemo.Util
{
public class PiiRemovalTextAnalyticsClientService : IPiiRemovalTextAnalyticsClientService
{
private TextAnalyticsClient _client;
public PiiRemovalTextAnalyticsClientService()
{
var azureEndpoint = Environment.GetEnvironmentVariable("AZURE_COGNITIVE_SERVICE_ENDPOINT");
var azureKey = Environment.GetEnvironmentVariable("AZURE_COGNITIVE_SERVICE_KEY");
if (string.IsNullOrWhiteSpace(azureEndpoint))
{
throw new ArgumentNullException(nameof(azureEndpoint), "Missing system environment variable: AZURE_COGNITIVE_SERVICE_ENDPOINT");
}
if (string.IsNullOrWhiteSpace(azureKey))
{
throw new ArgumentNullException(nameof(azureKey), "Missing system environment variable: AZURE_COGNITIVE_SERVICE_KEY");
}
_client = new TextAnalyticsClient(new Uri(azureEndpoint), new AzureKeyCredential(azureKey));
}
public async Task<Response<PiiEntityCollection>> RecognizePiiEntitiesAsync(string? document, string? language)
{
var piiEntities = await _client.RecognizePiiEntitiesAsync(document, language);
return piiEntities;
}
}
}
The UI codebehind of the razor component page showing the UI looks like this:
Home.razor.cs
using Azure;
using Microsoft.AspNetCore.Components;
using PiiDetectionDemo.Models;
using PiiDetectionDemo.Util;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace PiiDetectionDemo.Components.Pages
{
public partial class Home
{
private IndexModel Model = new();
private bool isProcessing = false;
private bool isSearchPerformed = false;
private async Task Submit()
{
isSearchPerformed = false;
isProcessing = true;
try
{
var response = await _piiRemovalTextAnalyticsClientService.RecognizePiiEntitiesAsync(Model.InputText, null);
Model.RedactedText = response?.Value?.RedactedText;
Model.UpdateHtmlRedactedText();
Model.AnalysisResult = response?.Value;
StateHasChanged();
}
catch (Exception ex)
{
await Console.Out.WriteLineAsync(ex.ToString());
}
isProcessing = false;
isSearchPerformed = true;
}
private void removeWhitespace(ChangeEventArgs args)
{
Model.InputText = args.Value?.ToString()?.CleanupAllWhiteSpace();
StateHasChanged();
}
}
}
To get the redacted or censored text void of any Pii that the Pii detection feature was able to detect, access the
Value of type Azure.AI.TextAnalytics.PiiEntityCollection. Inside this object, the string RedactedText contains the censored / redacted text.
The IndexModel looks like this :
using Azure.AI.TextAnalytics;
using Microsoft.AspNetCore.Components;
using PiiDetectionDemo.Util;
using System.ComponentModel.DataAnnotations;
using System.Text;
namespace PiiDetectionDemo.Models
{
public class IndexModel
{
[Required]
public string? InputText { get; set; }
public string? RedactedText { get; set; }
public string? HtmlRedactedText { get; set; }
public MarkupString HtmlRedactedTextMarkupString { get; set; }
public void UpdateHtmlRedactedText()
{
var sb = new StringBuilder(RedactedText);
if (AnalysisResult != null && RedactedText != null)
{
foreach (var piiEntity in AnalysisResult.OrderByDescending(a => a.Offset))
{
sb.Insert(piiEntity.Offset + piiEntity.Length, "</b></span>");
sb.Insert(piiEntity.Offset, $"<span style='background-color:lightgray;border:1px solid black;corner-radius:2px; color:{GetBackgroundColor(piiEntity)}' title='{piiEntity.Category}: {piiEntity.SubCategory} Confidence: {piiEntity.ConfidenceScore} Redacted Text: {piiEntity.Text}'><b>");
}
}
HtmlRedactedText = sb.ToString()?.CleanupAllWhiteSpace();
HtmlRedactedTextMarkupString = new MarkupString(HtmlRedactedText ?? string.Empty);
}
private string GetBackgroundColor(PiiEntity piiEntity)
{
if (piiEntity.Category == PiiEntityCategory.PhoneNumber)
{
return "yellow";
}
if (piiEntity.Category == PiiEntityCategory.Organization)
{
return "orange";
}
if (piiEntity.Category == PiiEntityCategory.Address)
{
return "green";
}
return "gray";
}
public long ExecutionTime { get; set; }
public PiiEntityCollection? AnalysisResult { get; set; }
}
}
Frontend UI looks like this: Home.razor
@page "/"
@using PiiDetectionDemo.Util
@inject IPiiRemovalTextAnalyticsClientService _piiRemovalTextAnalyticsClientService;
<h3>Azure HealthCare Text Analysis - Pii detection feature - Azure Cognitive Services</h3>
<em>Pii = Person identifiable information</em>
<EditForm Model="@Model" OnValidSubmit="@Submit">
<DataAnnotationsValidator />
<ValidationSummary />
<div class="form-group row">
<label><strong>Text input</strong></label>
<InputTextArea @oninput="removeWhitespace" class="overflow-scroll" style="max-height:500px;max-width:900px;font-size: 10pt;font-family:Verdana, Geneva, Tahoma, sans-serif" @bind-Value="@Model.InputText" rows="5" />
</div>
<div class="form-group row">
<div class="col">
<br />
<button class="btn btn-outline-primary" type="submit">Run</button>
</div>
<div class="col">
</div>
<div class="col">
</div>
</div>
<br />
@if (isProcessing)
{
<div class="progress" style="max-width: 90%">
<div class="progress-bar progress-bar-striped progress-bar-animated"
style="width: 100%; background-color: green">
Retrieving result from Azure Text Analysis Pii detection feature. Processing..
</div>
</div>
<br />
}
<div class="form-group row">
<label><strong>Analysis result</strong></label>
@if (isSearchPerformed)
{
<br />
<b>Execution time took: @Model.ExecutionTime ms (milliseconds)</b>
<br />
<br />
<b>Redacted text (Pii removed)</b>
<br />
<div class="form-group row">
<label><strong>Categorized Pii redacted text</strong></label>
<div>
@Model.HtmlRedactedTextMarkupString
</div>
</div>
<br />
<br />
<table class="table table-striped table-dark table-hover">
<thead>
<th>Pii text</th>
<th>Category</th>
<th>SubCategory</th>
<th>Offset</th>
<th>Length</th>
<th>ConfidenceScore</th>
</thead>
<tbody>
@if (Model.AnalysisResult != null) {
@foreach (var entity in Model.AnalysisResult)
{
<tr>
<td>@entity.Text</td>
<td>@entity.Category.ToString()</td>
<td>@entity.SubCategory</td>
<td>@entity.Offset</td>
<td>@entity.Length</td>
<td>@entity.ConfidenceScore</td>
</tr>
}
}
</tbody>
</table>
}
</div>
</EditForm>
The Demo uses Bootstrap 5 to build up a HTML table styled and showing the Azure.AI.TextAnalytics.PiiEntity properties.
