Importing Json File to SQL Server into a variable

A short article today of how to import JSON file to SQL Server into a variable, which can then
be used to insert it into a column of type NVARCHAR(MAX) of a table. The maximum size of NVARCHAR(MAX) is 2 Gb, so you can
import large Json files using this datatype. If the Json is small and below 4000 chars, use for example NVARCHAR(4000) instead. Here is a SQL script to import the json file using OPENROWSET and Bulk import. We also pass in the path to the folder where the json file is. It is put in the same folder as the .sql file script. Note that the variable $(FullScriptDir) is passed in via a .bat file (shown further below) and we expect the .json file to be in the same folder as the .bat file. You can provide a full path to a .json file instead and skip the .bat file here and import a json file, but it is nice to load the .json file from the same folder as the .sql file in case you want to copy the .sql and .json file to another server and not having to provide and possibly having to adjust the full path. Sql-script import_json_file_openrowset.sql:

DECLARE @JSONFILE VARCHAR(MAX); 

SELECT @JSONFILE = BulkColumn
FROM OPENROWSET (BULK '$(FullScriptDir)\top-posts.json', SINGLE_CLOB) AS j;

PRINT 'JsonFile contents: ' + @JSONFILE

IF (ISJSON(@JSONFILE)=1) PRINT 'It is valid Json';

The .bat file here passes the current folder as a variable to the sql script runsqlscript.bat

@set FullScriptDir=%CD%
sqlcmd -S .\SQLEXPRESS  -i import_json_file_openrowset.sql

This outputs:

sqlcmd -S .\SQLEXPRESS  -i import_json_file_openrowset.sql
JsonFile contents: [
   {
      "Id":6107,
      "Score":176,
      "ViewCount":155988,
      "Title":"What are deconvolutional layers?",
      "OwnerUserId":8820
   },
   {
      "Id":155,
      "Score":164,
      "ViewCount":25822,
      "Title":"Publicly Available Datasets",
      "OwnerUserId":227
   }
]
It is valid Json

With the variable JSONFILE you can do whatever with it such as inserting it to a column in a new row of a table for example.
Importing json from a string directly using OPENJSON

It is also possible to directly just import the JSON from a string variable like this:

DECLARE @JSONSTRINGSAMPLE VARCHAR(MAX) 

SET @JSONSTRINGSAMPLE = N'[
 {
    "Id": 2334,
    "Score": 4.3,
    "Title": "Python - Used as scientific tool for graphing"
 },
{
    "Id": 2335,
    "Score": 5.2,
    "Title": "C# : Math and physics programming"
 }
]';

SELECT * FROM OPENJSON (@JSONSTRINGSAMPLE) WITH (
    Id INT,
    Score REAL,
    Title NVARCHAR(100)
)

Functional programming - Fork combinator in C# to combine results from parts

This article will discuss a wellknown combinator called Fork which allows you to combine the mapped result. Consider the following extension methods to fork on an object. Fork here means to operate on parts of the object such as
different properties and apply functions on these parts and then recombine the results into a combined result via a specified combinator function, sometimes called a 'join function'.

public static class FunctionalExtensions {

	public static TOutput Map<TInput, TOutput>(
		this TInput @this,
		Func<TInput, TOutput> func) => func(@this);

	public static TOutput Fork<TInput, TMiddle, TOutput>(
		this TInput @this,
		Func<IEnumerable<TMiddle>, TOutput> combineFunc,
		params Func<TInput, TMiddle>[] parts)
	{
		var intermediateResults = parts.Select(p => p(@this));
		var result = combineFunc(intermediateResults);
		return result;
    }

	public static TOutput Fork<TInput, TMiddle, TOutput>(
		this TInput @this,
		Func<TInput, TMiddle> leftFunc,
		Func<TInput, TMiddle> rightFunc,
		Func<TMiddle, TMiddle, TOutput> combineFunc)
	{
		var leftResult = leftFunc(@this); // @this.Map(leftFunc);
		var rightResult = rightFunc(@this); // @this.Map(rightFunc);
		var combineResult = combineFunc(leftResult, rightResult);
		return combineResult;
	}

}

Let's take a familiar mathematical example, calculating the Hypotenuse in a triangle using Pythagorean theorem. This states that the length of the longest side A of a 'right triangle' is the square root of the sum of the squares of the shorter sides B and C : A = √(B² + C²) Consider this class:

  
  
  public class Triangle {
	public double CathetusA { get; set; }
	public double CathetusB { get; set; }	
	public double Hypotenuse { get; set; }
  }
  
    

Let's test the first Fork helper extension method accepting two functions for specifying the left and right components:

  
  
  	var triangle = new Triangle
	{
		CathetusA = 3,
		CathetusB = 4
	};
	
	triangle.Hypotenuse = triangle.Fork(	
		t => t.CathetusA * t.CathetusA, 
		t => t.CathetusB * t.CathetusB, 
		(l,r) => Math.Sqrt(l+r));
		
	Console.WriteLine(triangle.Hypotenuse);
  
  
  

This yields '5' as the answer via the forked result above. A simple example, but this allows us to create a simple combinatory logic example on an object of any type using functional programming (FP). Let's look at a simpler example just combining multiple properties of an object with a simple string-join, but using the Fork version supporting arbitrary number of parts / components:

 


public class Person {
	public string JobTitle { get; set; }
	public string FirstName { get; set; }
	public IEnumerable<string> MiddleNames { get; set; }
	public string LastName { get; set; }
}

var person = new Person{
		JobTitle = "Detective",
		FirstName = "Alexander",
		MiddleNames = new[] { "James", "Axel" },
		LastName = "Foley"
	};
	
string contactCardText = person.Fork(parts => string.Join(" ", parts), p => p.FirstName,
 p => string.Join(" ", p.MiddleNames), p => p.LastName);
Console.WriteLine(contactCardText);

This yields: Alexander James Axel Foley Fork can be very useful in many cases you need to 'branch off' on an object and recombine parts of the object with some specific function, either two parts or multiple parts and either continue to work on the results or retrieve the results.

Functional programming - the Tee function to inspect current state in a chained expression

In this article we will look at helper extension methods of StringBuilder first to better support chaining StringBuilder. We will work on the same StringBuilder instance and add support for appending lines or character to the StringBuilder given a condition. Also example showing how to aggregate lines from a sequence is shown and appending formatted lines. Since C# interpolation has become more easy to use, I would suggest you keep using AppendLine instead. Here is the helper methods in the extension class :

public static class StringBuilderExtensions {

	public static StringBuilder AppendSequence<T>(this StringBuilder @this, IEnumerable<T> sequence, Func<StringBuilder, T, StringBuilder> fn)
	{
		var sb = sequence.Aggregate(@this, fn);
		return sb;
	}
	
	public static StringBuilder AppendWhen(this StringBuilder @this, Func<bool> condition, Func<StringBuilder, StringBuilder> fn) => 
		condition() ? fn(@this) : @this;
		
    public static StringBuilder AppendFormattedLine(
		this StringBuilder @this,
		string format,
		params object[] args) => 
			@this.AppendFormat(format, args).AppendLine();
	
}

Now consider this example usage:

void Main()
{
	var countries = new Dictionary<int, string>{
		{ 1, "Norway" },
		{ 2, "France" },
		{ 3, "Austria" },
		{ 4, "Sweden" },
		{ 5, "Finland" },
		{ 6, "Netherlands" }
	};
	string options = BuildSelectBox(countries, "countriesSelect", true);
	options.Dump("Countries"); //dump is a method available in Linqpad to output objects 
	
}

private static string BuildSelectBox(IDictionary<int, string> options, string id, bool includeUnknown) =>
		new StringBuilder()
			.AppendFormattedLine($"<select id=\"{id}\" name=\"{id}\">")
			.AppendWhen(() => includeUnknown, sb => sb.AppendLine("\t<option value=\"0\">Unknown</option>"))
			.AppendSequence(options, (sb, item) => sb.AppendFormattedLine("\t<option value=\"{0}\">{1}</option>", item.Key, item.Value))
			.AppendLine($"</select>").ToString();   

What if we wanted to inspect the state of the stringbuilder in the middle of these chained expression. Is it possible to output state in such lengthy chained functional expressions? Yes, that is called the Tee method inside functional programming patterns. Other might call it for Tap such as used in Rx languages. The Tee method looks like this:

 
 
public static class FunctionalExtensions {

	public static T Tee<T>(this T @this, Action<T> act) {
		act(@this);
		return @this;
	}
	
}

 

We can now inspect state in the middle of chained expressions in functional expressions.

 
 
 
private static string BuildSelectBox(IDictionary<int, string> options, string id, bool includeUnknown) =>
		new StringBuilder()
			.AppendFormattedLine($"<select id=\"{id}\" name=\"{id}\">")
			.AppendWhen(() => includeUnknown, sb => sb.AppendLine("\t<option value=\"0\">Unknown</option>"))
            .Tee(Console.WriteLine)
			.AppendSequence(options, (sb, item) => sb.AppendFormattedLine("\t<option value=\"{0}\">{1}</option>", item.Key, item.Value))
			.AppendLine($"</select>").ToString();   
 
 

The picture below shows the output:

So there you have it, if you have lengthy chained functional expressions, make such a Tee helper method to peek into the state this far. The name Tee stems from the Unix Command by the same name. It copies contents from STDIN to STDOUT. More about Tee Unix command here:

https://shapeshed.com/unix-tee/

Functional programming - looking up current time and encapsulating usings

I looked at encapsulating Using statements today for functional programming and how to look up the current time with API available on the Internet.

public static class Disposable {
	
	public static TResult Using<TDisposable,TResult>(
		Func<TDisposable> factory,
		Func<TDisposable, TResult> map)		
		where TDisposable : IDisposable
	{
		using (var disposable = factory()){
			return map(disposable);
		}
		
	}	
}

void Main()
{
	var currentTime = EpochTime.AddSeconds(Disposable
			  .Using(() => new HttpClient(),
					client => JsonDocument.Parse(client.GetStringAsync(@"http://worldtimeapi.org/api/timezone/europe/oslo").Result))
			  .RootElement
			  .GetProperty("unixtime")
			 .GetInt64()).ToLocalTime(); //list of time zones available here: http://worldtimeapi.org/api/timezone
	currentTime.Dump("CurrentTime");	
}

public static DateTime EpochTime => new DateTime(1970, 1, 1);

The Disposable is abstracted away in the helper method called Using accepting a factory function to create a TDisposable that accepts an IDisposable. We look up the current time using the WorldTimeApi and make use of extracting the UnixTime which is measured from Epoch as the number of seconds elapsed from 1st January 1970. We make use of System.Text.Json here, which is part of .NET to parse the json retrieved.

Currying functions in C#

This article will look into helper methods for currying functions in C#. The definition of Currying consists of splitting up a function with multiple arguments into multiple functions accepting one argument. But you can also have some of the arguments provided via smaller functions, so be aware also of this alternative. What is in the name currying? The name has nothing to do with cooking from India, but comes from the mathematician Haskell Brooks Curry (!)

https://en.wikipedia.org/wiki/Haskell_Curry

A reason for introducing support for currying is that you can build complex functions from simpler functions as building blocks. Currying is explained great here:
https://www.c-sharpcorner.com/UploadFile/rmcochran/functional-programming-in-C-Sharp-currying/

We will see in the examples that we can provide multiple arguments at once and the syntax will look a bit special compared to other C# code. Curryings benefits is to allow a more flexible way to call a method. You can store into variables calls to a function providing a subset of argument and use that variable to either specify an intermediate other call or get the final result. Note - The function will be called when ALL arguments are provided ONCE ! This helps a lot of avoiding surprising side effects. Let's first look at a sample set of methods we want to support currying.

int FooFourArgs(string st, float x, int j, int k)
{
	Console.WriteLine($"Inside method FooFourArgs. Got parameters: st={st}, x={x}, j={j}, k={k}");
	return 42;
}

int FooThreeArgs(string st, float x, int j)
{
	Console.WriteLine($"Inside method FooThreeArgs. Got parameters: st={st}, x={x}, j={j}");
	return 42;
}

int FooTwoArgs(string st, float x)
{
	Console.WriteLine($"Inside method FooTwoArgs. Got parameters: st={st}, x={x}");
	return 41;
}

int FooOneArgs(string st)
{
	Console.WriteLine($"Inside method FooOneArgs. Got parameters: st={st}");
	return 40;
}

We want to call the sample methods above in a more flexible way by splitting the number of arguments we provide. Let's see the extension methods to call up to four arguments to a function. Note the use of chaining the lambda operator (=>) to provide the support for currying.

public static class FunctionExtensions
{
	public static Func<T1, TResult> Curried<T1, TResult>(this Func<T1, TResult> func)
	{
		return x1 => func(x1);
	}
	
	public static Func<T1, Func<T2, TResult>> Curried<T1, T2, TResult>(this Func<T1, T2, TResult> func)
	{
		return x1 => x2 => func(x1, x2);
	}

	public static Func<T1, Func<T2, Func<T3, TResult>>> Curried<T1, T2, T3, TResult>(this Func<T1, T2, T3, TResult> func)
	{
		return x1 => x2 => x3 => func(x1, x2, x3);
	}

	public static Func<T1, Func<T2, Func<T3, Func<T4, TResult>>>> Curried<T1, T2, T3, T4, TResult>(this Func<T1, T2, T3, T4, TResult> func)
	{
		return x1 => x2 => x3 => x4 => func(x1, x2, x3,x4);
	}
}

The following main method shows how to use these curry helper methods:

void Main()
{
	var curryOneArgsDelegate = new Func<string, int>((st) => FooOneArgs(st)).Curried();
	var curryOneArgsPhaseOne = curryOneArgsDelegate("hello");

	var curryTwoArgsDelegate = new Func<string, float, int>((st, x) => FooTwoArgs(st,x)).Curried();
	var curryTwoArgsPhaseOne = curryTwoArgsDelegate("hello");
	var curryTwoArgsPhaseTwo = curryTwoArgsPhaseOne(3.14f);

	var curryThreeArgsDelegate = new Func<string, float, int, int>((st, x, j) => FooThreeArgs(st, x, j)).Curried();
	var curryThreeArgsPhaseOne = curryThreeArgsDelegate("hello");
	var curryThreeArgsPhaseTwo = curryThreeArgsPhaseOne(3.14f);
	var curryThreeArgsPhaseThree = curryThreeArgsPhaseTwo(123);	
	//Or call currying in a single call passing in two or more parametres
	var curryThreeArgsPhaseOneToThree = curryThreeArgsDelegate("hello")(3.14f)(123);

	var curryFourArgsDelegate = new Func<string, float, int, int, int>((st, x, j, k) => FooFourArgs(st, x, j, k)).Curried();
	var curryFourArgsPhaseOne = curryFourArgsDelegate("hello");
	var curryFourArgsNextPhases = curryFourArgsPhaseOne(3.14f)(123)(456); //just pass in the last arguments if they are known at this stage
	curryFourArgsDelegate("hello")(3.14f)(123)(456); //you can pass in 1-4 parameters to FooFourArgs method - all in a single call for example or one by one
}

The output we get is this. Note that we only call the methods we defined when all parameters are sent in. The function call which had partial argument list provided did not result into a function call.

Inside method FooOneArgs. Got parameters: st=hello
Inside method FooTwoArgs. Got parameters: st=hello, x=3,14
Inside method FooThreeArgs. Got parameters: st=hello, x=3,14, j=123
Inside method FooThreeArgs. Got parameters: st=hello, x=3,14, j=123
Inside method FooFourArgs. Got parameters: st=hello, x=3,14, j=123, k=456

So from a higher level, currying a function f(x,y,z) means adding support that you could call the function like this:
f(x,g(y,z)) or f(x,g(y,h(z))) - there more arguments you get there is more variations of number of parameters and methods you can pass in. Here is another example how you can build up a calculation uing simpler methods.

void Main()
{
	Func Area = (x,y) => x*y;
	Func CubicArea = (x,y,z) => Area.Curried()(Area(x,y))(z);	
	CubicArea(3,2,4); //supplying all arguments manully is okay
}

CubicArea expects THREE arguments. The implementation allows us to use the Area function and via currying we can use that method and provide the last third argument avoiding compilation error. Currying makes your functions allow more flexible ways of being called.

Using IronPython to execute Python code from .NET

Let's look at some code showing how to execute Python code from .NET using IronPython! IronPython provides support for Python scripts to run inside .NET and utilizes the Dynamic Language Runtime - DLR. The DLR together allows the caller to get dynamic typing and dynamic method dispatch, which is central in the dynamic languages such as Python. IronPython was first released in 2004, some 20 years ago. It has continued to evolve slowly and provides seamless integration into .NET ecosystem for Python developers. In this article, I will present some simple code that shows how you can run Python code inside a .NET 8 console application. We will load up some tuples in an array in some simple Python code, using IronPython. Tuples in Python Tuples in Python are immutable (such as in C#) and are defined using parentheses and comma-separated. This is the same as in C#, but Python had tuple support over 20 years before C#. We will have to add one Nuget package, the IronPython package, in a net8.0 application.

HelloIronPythonDemo1.csproj

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="IronPython" Version="3.4.1" />
  </ItemGroup>

  <ItemGroup>
    <None Update="customers.py">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
  </ItemGroup>

</Project>

Consider the following array of tuples in Python :

customers.py

customers = [
	('Jenna', 42, 165),
	('Thor', 40, 174),
	('Christopher', 18, 170),
	('Liz', 16, 168),
 ]

Python code is very compact and you declare variables without specifying type such as in C#, Python uses a simple way of creating variables and while C# got support in C# 7 in 2017, Python has had support for tuples since its early days. In the Python 1.4 version, we find it documented here:
https://docs.python.org/release/1.4/tut/node37.html#SECTION00630000000000000000.
Bear in mind, this is way back in 1996, C# was over 20 years later with its tuple support. If you install IronPython, you get a terminal where you can enter Python code (plus more functionality with .NET) such as shown below, where tuples are created and tuples may be composed or 'packed' and also 'unpacked', which is called deconstructed in .NET tuples.

To execute code to retrieve this array of tuples, first create a ScriptEngine and then create a ScriptScope, which we will use to retrieve the Python-declared variable customers. We create a ScriptSource, where we use the ScriptEngine to load up either a string or a file. A dynamic variable will be used to get the array of tuples and we can loop through this array with a foreach loop and output its content.

Program.cs

 
 using IronPython.Hosting;
 using Microsoft.Scripting.Hosting;
 using static System.Console;   
 IronPythonDemo1.OutputSomeExternallyLoadedTuples();

 public class IronPythonDemo1
 {

    public static void OutputSomeExternallyLoadedTuples()
    {
        var engine = Python.CreateEngine();
        ScriptScope scope = engine.CreateScope();

        //ScriptSource source = engine.CreateScriptSourceFromString(tupleStatement);
        ScriptSource source = engine.CreateScriptSourceFromFile("customers.py");
        source.Execute(scope);
        dynamic customers = scope.GetVariable("customers");
        foreach (var customer in customers)
        {
            Console.WriteLine($"(Name = {StringExtensions.FixedLength(customer[0], 20)}, Age = {StringExtensions.FixedLength(customer[1].ToString(), 8)}, Height={StringExtensions.FixedLength(customer[2].ToString(), 8)})");
        }
    }

}
 
 
 

Documentation for named tuples are available here: https://docs.python.org/3/library/collections.html#collections.namedtuple Here is sample coding showing script that although it is more verbose, shows more readability of which field is which for a named tuple. In an ordinary tuple, you use indexes to retrieve the nth field (0-based). But with named tuples, you use a field name instead.

 
from collections import namedtuple

Customer = namedtuple('Customer', ['Name', 'Age', 'Height'])
 
customers2 =  [
    Customer(Name = 'Jenna', Age = 42, Height = 165),
    Customer(Name = 'Thor', Age = 38, Height = 174),
    Customer(Name = 'Christopher', Age = 42, Height = 170),
    Customer(Name = 'Liz', Age = 42, Height = 168),
 ]
   
for cust in customers2:
    print(f"{cust.Name} with a height of {cust.Height}(cm)")

This outputs:

 
Jenna with a height of 165(cm)
Thor with a height of 174(cm)
Christopher with a height of 170(cm)
Liz with a height of 168(cm)
When your tuple gets many fields, having this readability should reduce bugs. Also, if you add more fields to your tuple, you do not have to fix up indexes in your script. So code is a bit more verbose, but it is also more open for change and readable.

The FixedLength extension method is a simple method to output text to a fixed width.

 
 
 public static class StringExtensions
 {

    public static string FixedLength(this string input, int length, char paddingchar = ' ')
    {
        if (string.IsNullOrWhiteSpace(input))
        {
            return input;
        }
        if (input.Length > length)
            return input.Substring(0, length);
        else
            return input.PadRight(length, paddingchar);
    }

 } 
 

Creating a data table from IEnumerable of T and defining column order explicitly in C#

This article shows code how you can create a DataTable from a collection of T (IEnumerable<T>) and defining explicitly the column order. An extension method for this looks like the following:

public static class DataTableExtensions
{


	public static DataTable CreateOrderedDataTable<T>(this IEnumerable<T> data)
	{
		var dataTable = new DataTable();
		var orderedProps = typeof(T).GetProperties(BindingFlags.Instance | BindingFlags.Public)
		 .OrderBy(prop => GetColumnOrder(prop)).ToList();
		
		foreach (var prop in orderedProps){
			dataTable.Columns.Add(prop.Name, Nullable.GetUnderlyingType(prop.PropertyType) ?? prop.PropertyType);
		}
		
		if (data != null)
		{
			dataTable.BeginLoadData();
			var enumerator = data.GetEnumerator();
			while (enumerator.MoveNext()){
			   var item = enumerator.Current;
			   var rowValues = new List<object>();
			   foreach (var prop in orderedProps){
			    rowValues.Add(prop.GetValue(item, null));		   	
			   }
			   dataTable.Rows.Add(rowValues.ToArray());			 			
			}
			dataTable.AcceptChanges();
		}
		return dataTable;
	}

	static int GetColumnOrder(PropertyInfo prop)
	{
		var displayAttribute = prop.GetCustomAttributes(typeof(DisplayAttribute), false).FirstOrDefault() as DisplayAttribute;
		int orderKey = displayAttribute?.Order ?? prop.MetadataToken;		
		return orderKey;
	}
	
}

We order first by DisplayAttribute and the Order value, and fallback to property's MetadataToken. This is an integer value that also returns the order the property was declared, in case you want to order just by the way properties are defined. We get the enumerator here and fetch the row one by one. We could use a simple foreach loop here too. Note the use of BeginLoadData and AcceptChanges. Consider the two classes next. One class does not set any explicit order, the other class uses the Display attribute's Order value to define a custom order of columns for the DataTable.

public class Car
{

	public int Id { get; set; }

	public string Make { get; set; }

	public string Model { get; set; }

	public string Color { get; set; }
}


public class CarV2
{
	[Display(Order = 4)]
	public int Id { get; set; }
	
	[Display(Order = 3)]
	public string Make { get; set; }
	
	[Display(Order = 2)]
	public string Model { get; set; }

	[Display(Order = 14)]
	public bool IsElectric { get; set; }

	[Display(Order = -188865)]
	public string Color { get; set; }
	
}

Next, the following little program in Linqpad tests this extension method and displays the datatables resulting with column ordering set.

void Main()
{
	var cars = new List<Car>{
		new Car { Id = 1, Make = "Audi", Model = "A5", Color = "Blue" },
		new Car { Id = 2, Make = "Volvo", Model = "XC60", Color = "Silver" },
		new Car { Id = 3, Make = "Golf", Model = "GTI", Color = "White" },
		new Car { Id = 4, Make = "Audi", Model = "A5", Color = "Blue" },
	};
	var dataTable = cars.CreateOrderedDataTable();
	dataTable.Dump("Cars datatable, data type is: Car");
	
	var carV2s = new List<CarV2>{
		new CarV2 { Id = 1, Make = "Audi", Model = "A5", Color = "Blue" },
		new CarV2 { Id = 2, Make = "Volvo", Model = "XC60", Color = "Silver" },
		new CarV2 { Id = 3, Make = "Golf", Model = "GTI", Color = "White" },
		new CarV2 { Id = 4, Make = "Audi", Model = "A5", Color = "Blue" },
	};	
	var dataTableV2 = carV2s.CreateOrderedDataTable();
	dataTableV2.Dump("Carsv2 datatable, datatype is CarV2");

}

Generating repeated data into variable in SQL Server in T-SQL

Let's see how we can create repeated data into variable of SQL Server in T-SQL. Use the REPLICATE function to create repeated data like this:

DECLARE @myVariable NVARCHAR(MAX)
SET @myVariable = REPLICATE('.', 10)
PRINT @myVariable
PRINT len(@myVariable)

In case you want to set the variable to data which is longer than 8000 characters, you must convert the argument to NVARCHAR(MAX).

DECLARE @myVariable NVARCHAR(MAX)
SET @myVariable = REPLICATE(CONVERT(NVARCHAR(MAX),'.'), 1024*1024*2)
PRINT len(@myVariable)

Creating random content is also easy in T-SQL:

DECLARE @myVariable NVARCHAR(MAX)
SET @myVariable = REPLICATE(CONVERT(NVARCHAR(MAX),REPLACE(NEWID(),'-', '')), 4)
PRINT len(@myVariable)
PRINT @myVariable

NEWID() creates a new guid, and we strip away the '-' letter, giving 32 chars which we replicate above four times. Since we were below 8000 chars, we chould have skipped using convert to nvarchar(max).

Password hashing in .NET

This article will look on different ways to hash a password in .NET. MD5 was developed by Ron Rivest in 1991 and was used a lot in the 90s, but in 2005 it was revealed it contains collisions. MD5 and SHA-1 is not advised to used in sensitive hashing related to security anymore. Instead, a PBKDF or Password Derived Key-derivation function algorithm will be used. A PBKDF2-based method in Rfc2898DeriveBytes will be used. It has been available since .NET 6. Users of Asp.net Core Identity are recommended to use PasswordHasher instead : https://andrewlock.net/exploring-the-asp-net-core-identity-passwordhasher/ An overview of the arithmetic flow of PBKDF2 is shown below. In the diagram, SHA-512 is indicated, but the code shown in this article
uses SHA-256.

First off, to do a MD5 hash we can use the following :

 
 
 static string Md5(string input){
	using (var md5 = MD5.Create()){
		var byteHash = md5.ComputeHash(Encoding.UTF8.GetBytes(input)); 
		var hash = BitConverter.ToString(byteHash).Replace("-", "");
		return hash;
	}
}
 
 

And to test it out we can run the following:

 
 
 void Md5Demo()
{
	string inputPassword = "abc123";
	string md5Hash = Md5(inputPassword);
	Console.WriteLine("MD5 Demonstration in .NET");
	Console.WriteLine("-------------------------");
	Console.WriteLine($"Password to hash: {inputPassword}");
	Console.WriteLine($"MD5 hashed password: {md5Hash}");
	Console.WriteLine();
} 
 
 

MD5 Demonstration in .NET ------------------------- Password to hash: abc123 MD5 hashed password: E99A18C428CB38D5F260853678922E03 The MD5 hash above agrees with the online MD5 hash here: https://www.md5hashgenerator.com/ MD5 method here does not mention any salt, but this could be concatenated with the password to prevent against rainbow table attacks, that is dictionary attacks. Next, to perform PDKDF2 hashing, the code below can be used. Note that this algorithm will be run iteratively to generate a hash value that is increasingly more computationally expensive to calculate the hash of compared to the number of iterations and includes a salt, making it scalable
to be more and more difficult for attacks.

static byte[] _salt = RandomNumberGenerator.GetBytes(32);

static void HashPassword(string passwordToHash, int numberOfRounds)
{
	var sw = Stopwatch.StartNew();
	var hashedPassword = Rfc2898DeriveBytes.Pbkdf2(
		passwordToHash,
		_salt,
		numberOfRounds,
		HashAlgorithmName.SHA256,
		32);
	sw.Stop();

	Console.WriteLine();
	Console.WriteLine("Password to hash : " + passwordToHash);
	Console.WriteLine("Hashed Password : " + Convert.ToBase64String(hashedPassword));
	Console.WriteLine("Iterations < " + numberOfRounds + "> Elapsed Time: " + sw.ElapsedMilliseconds + " ms");
}

The value 32 here is the desired output length of the hash, we can decide how long the hash we get out of the call to the method. We can then test out the Pbkdf2 method using an increasing number of iterations.

 
 
 void RunPbkdf2HashDemo()
{
	const string passwordToHash = "abc123";

	Console.WriteLine("Password Based Key Derivation Function Demonstration in .NET");
	Console.WriteLine("------------------------------------------------------------");
	Console.WriteLine();
	Console.WriteLine("PBKDF2 Hashes using Rfc2898DeriveBytes");
	Console.WriteLine();

	HashPassword(passwordToHash, 1);
	HashPassword(passwordToHash, 10);
	HashPassword(passwordToHash, 100);
	HashPassword(passwordToHash, 1000);
	HashPassword(passwordToHash, 10000);
	HashPassword(passwordToHash, 100000);
	HashPassword(passwordToHash, 1000000);
	HashPassword(passwordToHash, 5000000);
} 
 
 

This gives the following output:

 
 
Password Based Key Derivation Function Demonstration in .NET
------------------------------------------------------------

PBKDF2 Hashes using Rfc2898DeriveBytes

Password to hash : abc123
Hashed Password : eqeul5z7l2dPrOo8WjH/oTt0RYHvlZ2lvk8SUoTjZq4=
Iterations (1) Elapsed Time: 0 ms

Password to hash : abc123
Hashed Password : wfd8qQobzBPZvdemqrtZczqctFe0JeAkKjU3IJ48cms=
Iterations (10) Elapsed Time: 0 ms

Password to hash : abc123
Hashed Password : VY45SxzhqjYronha0kt1mQx+JRDVlXj82prX3H7kjII=
Iterations (100) Elapsed Time: 0 ms

Password to hash : abc123
Hashed Password : B0LfHgRSslG/lWe7hbp4jb8dEqQ/bZwNtxsaqbVBZ2I=
Iterations (1000) Elapsed Time: 0 ms

Password to hash : abc123
Hashed Password : LAHwpS4bnbO7CQ1r7buYgUTrp10FyaRyeK6hCwGwv20=
Iterations (10000) Elapsed Time: 1 ms

Password to hash : abc123
Hashed Password : WDjyPySpULXtVOVmSR9cYlzAY4LWeJqDBhszKAfIaPc=
Iterations (100000) Elapsed Time: 13 ms

Password to hash : abc123
Hashed Password : sDx6sOrTl2b7cNZGUAecg7YO4Md/g3eAtfQSvh/vxpM=
Iterations (1000000) Elapsed Time: 127 ms

Password to hash : abc123
Hashed Password : ruywLaR0QApOU5bkqE/x2AAhYJzBj5y6D3P3IxlIF2I=
Iterations (5000000) Elapsed Time: 643 ms
 
 

Note that it takes many iterations before the computation takes significant time. Sources / links :

• OWASP recommends using 600 000 iterations when using SHA256 as noted here: https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html#pbkdf2
• Hashing passwords using PBKDF2 are explained by Stephen Haunts here: Hashing Passwords Safely using a Password Based Key Derivation Function (PBKDF2)

AES Encryption with Galois Counter Mode (GCM) in C#

This article presents some helper methods for performing AES Encryption using Galois Counter Mode (GCM). AES or Advanced Encryption Standard is the most used encryption algorithm used today, having overtaken DES and Triple DES since 2001. We will look into the GCM mode of AES in this article. AES-GCM class AesGcm is supported in .NET Core 3.0 and newer .NET versions, plus in .NET Standard 2.1. AES-GCM is authenticated encryption, compared to default AES-CBC (Cipher Block Chaining). Benefits of using GCM mode of AES is the following:

• Data authenticity / integrity. This is provided via a tag that is outputted by the encryption and used while decrypting
• Provides support for sending additional data, used for example in newer TLS implementations to provide both encryption and a non-encrypted payload. This is called additional metadata

Here is a helper class to perform encryption and decryption using AES-GCM.

 
 public static class AesGcmEncryption {


	public static (byte[], byte[]) Encrypt(byte[] dataToEncrypt, byte[] key, byte[] nonce, byte[] associatedData = null)
	{
		using var aesGcm = new AesGcm(key);
		//tag and ciphertext will be filled during encryption
		var tag = new byte[16]; //tag is a hmac (hash-based message authentication code) to check that information has not been tampered with
	    var cipherText = new byte[dataToEncrypt.Length];
		aesGcm.Encrypt(nonce, dataToEncrypt, cipherText, tag, associatedData);
		return (cipherText, tag);
	}

	public static byte[] Decrypt(byte[] cipherText, byte[] key, byte[] nonce, byte[] tag, byte[] associatedData = null)
	{
		using var aesGcm = new AesGcm(key);
		//tag and ciphertext will be filled during encryption
		var decryptedData = new byte[cipherText.Length];
		aesGcm.Decrypt(nonce, cipherText, tag, decryptedData, associatedData);
		return decryptedData;
	}
	
}
 
 

In the code above, the encrypt method returns a tuple with the ciperText and the tag. These are the encrypted data and the tag, both must be used while decrypting and the tag provides as mentioned a means of checking the integrity of data, i.e. that data has not been tampered with. Note that the 16-byte tag and the ciphertext is filled after running the Encrypt method of the AesGcm class. The cipherText array must be the same length as the dataToEncrypt array inputted. Here is sample code to use AES-GCM. Note that the metadata used here, while optional, must match in case it is set in the encryption and decryption. The nonce must be 12 bytes - 96 bits in length.The nonce is similar to a initialization vector, although it is used once for the particular encryption and decryption, it is used to protect against replay attacks.

 
 
 void TestAesGCM()
{
	const string original = "Text to encrypt";
	var key = RandomNumberGenerator.GetBytes(32); //256 bits key
	var nonce = RandomNumberGenerator.GetBytes(12); //96 bits nonce
	
	(byte[] cipherText, byte[] tag) result = AesGcmEncryption.Encrypt(Encoding.UTF8.GetBytes(original),
	 key, nonce, Encoding.UTF8.GetBytes("some metadata 123"));
	 byte[] decryptedText = AesGcmEncryption.Decrypt(result.cipherText, key, nonce, result.tag, Encoding.UTF8.GetBytes("some metadata 123")); 
		
	Console.WriteLine("AES Encryption demo GCM - Galois Counter Mode:");
	Console.WriteLine("--------------");
	Console.WriteLine("Original Text = " + original);
	Console.WriteLine("Encrypted Text = " + Convert.ToBase64String(result.cipherText));
	Console.WriteLine("Tag = " + Convert.ToBase64String(result.tag));
	Console.WriteLine("Decrypted Text = " + Encoding.UTF8.GetString(decryptedText));
}
 
 

AES Encryption demo GCM - Galois Counter Mode: -------------- Original Text = Text to encrypt Encrypted Text = 9+2x0kctnRwiDDHBm0/H Tag = sSDxsg17HFdjE4cuqRlroQ== Decrypted Text = Text to encrypt Use AES-GCM to provide integrity checking and allowing to send in metadata if desired to encrypt and decrypting with the AES algorithm. We can protect the AES key using different methods, for example using the Data Protection API, this is only supported in Windows. Let's look at a helper class for using Data Protection API.

 
 
 public static class DataProtectionUtil {

	public static byte[] Protect(byte[] dataToEncrypt, byte[] optionalEntropy, DataProtectionScope scope)
	{
		var encryptedData = ProtectedData.Protect(dataToEncrypt, optionalEntropy, scope);
		return encryptedData;
	}
	
	public static byte[] Unprotect(byte[] encryptedData, byte[] optionalEntropy, DataProtectionScope scope){
		var decryptedData = ProtectedData.Unprotect(encryptedData, optionalEntropy, scope);
		return decryptedData;
	}

	public static string Protect(string dataToEncrypt, string optionalEntropy, DataProtectionScope scope)
	{
		var encryptedData = ProtectedData.Protect(Encoding.UTF8.GetBytes(dataToEncrypt), optionalEntropy != null ? Encoding.UTF8.GetBytes(optionalEntropy) : null, scope);
		return Convert.ToBase64String(encryptedData);
	}

	public static string Unprotect(string encryptedData, string optionalEntropy, DataProtectionScope scope)
	{
		var decryptedData = ProtectedData.Unprotect(Convert.FromBase64String(encryptedData), optionalEntropy != null ? Encoding.UTF8.GetBytes(optionalEntropy) : null, scope);
		return Encoding.UTF8.GetString(decryptedData);
	}

}
 
 

An example how to protect your AES key:

 
 
void EncryptAndDecryptWithProtectedKey(){
	var original = "Text to encrypt";
	Console.WriteLine($"Original Text = {original}");
	
	//Create key and nnoce . Encrypt our text with AES 
	var gcmKey = RandomNumberGenerator.GetBytes(32);
	var nonce = RandomNumberGenerator.GetBytes(12); 
	
	var result = EncryptText(original, gcmKey, nonce); 
	
	//Create some entropy and protect AES key
	var entropy = RandomNumberGenerator.GetBytes(16); 
	var protectedKey = ProtectedData.Protect(gcmKey, entropy, DataProtectionScope.CurrentUser);

	Console.WriteLine($"gcmKey = {Convert.ToBase64String(gcmKey)}, protectedKey = {Convert.ToBase64String(protectedKey)}");
	
	// Decrypt the text with AES. the AES key has to be retrieved with DPAPI.
	var decryptedText = DecryptText(result.encrypted, nonce, result.tag, protectedKey, entropy);

	Console.WriteLine($"Decrypted Text using AES GCM with key retrieved via Data Protection API = {decryptedText}");

}

private static (byte[] encrypted, byte[] tag) EncryptText(string original, byte[] gcmKey, byte[] nonce){
	return AesGcmEncryption.Encrypt(Encoding.UTF8.GetBytes(original), gcmKey, nonce, Encoding.UTF8.GetBytes("some meta"));
}

private static string DecryptText(byte[] encrypted, byte[] nonce, byte[] tag, byte[] protectedKey, byte[] entropy){
	
	var key = DataProtectionUtil.Unprotect(protectedKey, entropy, DataProtectionScope.CurrentUser);

	Console.WriteLine($"Inside DecryptText: gcmKey = {Convert.ToBase64String(key)}, protectedKey = {Convert.ToBase64String(protectedKey)}");

	var decryptedText = AesGcmEncryption.Decrypt(encrypted, key, nonce, tag, Encoding.UTF8.GetBytes("some meta"));
	return Encoding.UTF8.GetString(decryptedText);
}
 

Data Protection API is only supported on Windows platform, there are more possibilities to protect AES key but protecting your key is always a challenge when dealing with symmetric encryption algorithms such as AES. Some more links:

• What is a Nonce ? https://www.youtube.com/watch?v=EOgpr73-pgc
• AES Explained - https://www.youtube.com/watch?v=O4xNJsjtN6E
• Galois/Counter Mode (GCM) - https://www.youtube.com/watch?v=V2TlG3JbGp0
• Authenticated Encryption in .NET with AES-GCM - https://www.scottbrady91.com/c-sharp/aes-gcm-dotnet

Digital signatures with RSA in .NET

I have looked at Digital signatures with RSA in .NET today. Digital signatures are used to provide non-repudiation, an authenticity proof that the original sender is who the sender claims to be and also that the data has not been hampered with. We will return a tuple of both a SHA-256 computed hash of some document data and also its digital signature using the RSA algorithm. I have used .netstandard 2.0 here, so the code can be used in most frameworks in both .NET Framework and .NET. We will use RSA here to do the digital signature signing and verification. First off, here is a helper class to create a RSA encrypted signature of a SHA-256 hash, here we create a new RSA with key size 2048. RsaDigitalSignature.cs

 
 
 public class RsaDigitalSignature
{
	private RSA _rsa;


	public RsaDigitalSignature()
	{
		_rsa = RSA.Create();
		_rsa.KeySize = 2048;
	}
	
	public static byte[] ComputeHashSha256(byte[] toBeHashed)
	{
		using (var sha256 = SHA256.Create())
		{
			return sha256.ComputeHash(toBeHashed);
		}
	}

	public (byte[] Signature, byte[] HashOfData) SignData(byte[] dataToSign)
	{
		var hashOfDataToSign = ComputeHashSha256(dataToSign);
		return (_rsa.SignHash(
			hashOfDataToSign,
			HashAlgorithmName.SHA256,
			RSASignaturePadding.Pkcs1),
			hashOfDataToSign);
	}

	public bool VerifySignature(byte[] signature, byte[] hashOfDataToSign)
	{
		return _rsa.VerifyHash(hashOfDataToSign, signature, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1);
	}

}
 
 
 

In the code above, we receive some document data and create the SHA-255 hash, which is computed. We return a tuple with the signed hash from the computed SHA-256 hash and also the computed SHA-256 hash itself. A console application that runs the sample code above is the following:

 
 
 void Main()
{
	SignAndVerifyData();
	//Console.ReadLine();
}

private static void SignAndVerifyData()
{
	Console.WriteLine("RSA-based sigital signature demo");
	var document = Encoding.UTF8.GetBytes("Document to sign");	
	var digitalSignature = new RsaDigitalSignature();
	var signature = digitalSignature.SignData(document);
	bool isValidSignature = digitalSignature.VerifySignature(signature.Signature, signature.HashOfData);
	Console.WriteLine($"\nInput Document:\n{Convert.ToBase64String(document)}\nIs the digital signature valid? {isValidSignature} \nSignature: {Convert.ToBase64String(signature.Signature)} \nHash of data:\n{ Convert.ToBase64String(signature.HashOfData)}");
}
 
 

Our verification of the signature shows that the verification of the digital signature passes.

 
Input Document:
RG9jdW1lbnQgdG8gc2lnbg==
Is the digital signature valid? True
Signature: Gok1x8Wxm9u5jTRcqrgPsI45ie3WPZLi/FNbaJMGTHqBmNbpJTEYjsXix97aIF6uPjgrxQWJKCegc8S4yASdut7TpJafO9wSRqvScc2SuOGK9BqnX+9GwRRQNti8ynm0ARRar+Z4hTpYY/XngFZ+ovvqIT3KRMK/7tsMmTg87mY0KelteFX7z7G7wPB9kKjT6ORYK4lVr35fihrbxei0XQP59YuEdALy+vbvKUm3JNv4sBU0lc9ZKpp2XF0rud8UnY1Nz4/XH7ZoaKfca5HXs9yq89DJRaPBRi1+Wv41vTCf8zFKPWZJrw6rm6kBMNHMENYbeBNdZyiCspTsHZmsVA==
Hash of data:
VPPxOVW2A38lCB810vuZbBH50KQaPSCouN0+tOpYDYs=
 

The code above uses a RSA created on the fly and is not so easy to share between a sender and a receiver. Let's look at how we can use X509 certificates to do the RSA encyption. It should be possible to share the source code below between the sender and the receiver and for example
export the public part of the X509 certificate to the receiver, which the receiver could install in a certificate store, only requred to know the thumbprint of the cert which is easy to see in MMC (Microsoft Management Console) or using Powershell and cd-ing into cert:\ folder . Let's first look at a helper class to get hold of a installed X509 certificate.

public class CertStoreUtil
{
	public static System.Security.Cryptography.X509Certificates.X509Certificate2 GetCertificateFromStore(
	System.Security.Cryptography.X509Certificates.StoreLocation storeLocation,
	string thumbprint, bool validOnly = true) {
	 var store = new X509Store(storeLocation);
	 store.Open(OpenFlags.ReadOnly);
	 var cert = store.Certificates.Find(X509FindType.FindByThumbprint, thumbprint, validOnly).FirstOrDefault();
	 store.Close();
	 return cert;
	}
}

Next up, a helper class to create a RSA-based digital signature like in the previous example, but using a certificate.

 
 
 public class RsaFromCertDigitalSignature
{
	private RSA _privateKey;
	private RSA _publicKey;

	public RsaFromCertDigitalSignature(StoreLocation storeLocation, string thumbprint)
	{
		_privateKey = CertStoreUtil.GetCertificateFromStore(StoreLocation.LocalMachine, thumbprint).GetRSAPrivateKey();
		_publicKey = CertStoreUtil.GetCertificateFromStore(StoreLocation.LocalMachine, thumbprint).GetRSAPrivateKey();
	}

	public static byte[] ComputeHashSha256(byte[] toBeHashed)
	{
		using (var sha256 = SHA256.Create())
		{
			return sha256.ComputeHash(toBeHashed);
		}
	}

	public (byte[] Signature, byte[] HashOfData) SignData(byte[] dataToSign)
	{
		var hashOfDataToSign = ComputeHashSha256(dataToSign);
		return (_privateKey.SignHash(
			hashOfDataToSign,
			HashAlgorithmName.SHA256,
			RSASignaturePadding.Pkcs1),
			hashOfDataToSign);
	}

	public bool VerifySignature(byte[] signature, byte[] hashOfDataToSign)
	{
		return _publicKey.VerifyHash(hashOfDataToSign, signature, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1);
	}

}

 
 

A console app that tests out the code above is shown next, I have selected a random cert on my dev pc here.

 
 
 void Main()
{
	SignAndVerifyData();
	//Console.ReadLine();
}

private static void SignAndVerifyData()
{
	Console.WriteLine("RSA-based sigital signature demo");
	var document = Encoding.UTF8.GetBytes("Document to sign");

	//var x509CertLocalHost = CertStoreUtil.GetCertificateFromStore(StoreLocation.LocalMachine, "1f0b749ff936abddad89f4bbea7c30ed64e3dd07");
		
	var digitalSignatureWithCert = new RsaFromCertDigitalSignature(StoreLocation.LocalMachine, "1f0b749ff936abddad89f4bbea7c30ed64e3dd07");
	var signatureWithCert = digitalSignatureWithCert.SignData(document);
	bool isValidSignatureFromCert = digitalSignatureWithCert.VerifySignature(signatureWithCert.Signature, signatureWithCert.HashOfData);
    Console.WriteLine(
		$@"Input Document:
		{Convert.ToBase64String(document)}
		Is the digital signature signed with private key of CERT valid according to public key of CERT? {isValidSignatureFromCert}
		Signature: {Convert.ToBase64String(signatureWithCert.Signature)} 
		Hash of data:\n{Convert.ToBase64String(signatureWithCert.HashOfData)}");
}

 
 

Now here is an important concept in digital signatures :

• For digital signatures, we MUST use a private key (e.g. private key of RSA instance, which can either be made on the fly or retrieved from for example a X509 certificate. Or a Json web key in a more modern example.
• For digital signature, to verify a signature we can use either the public or the private key, usually just the public key (which can be shared). For X509 certiifcates, we usually share a public cert (.cert or similar format) and keep our private cert ourselves (.pfx).

Sample output of the console app shown above:

 
 RSA-based sigital signature demo
Input Document:
    RG9jdW1lbnQgdG8gc2lnbg==
    Is the digital signature signed with private key of CERT valid according to public key of CERT? True
    Signature: ZHWzJeZnwbfI109uK0T4ubq4B+CHedQPIDgPREz+Eq9BR6A9y6kQEvSrxqUHvOppSDN5kDt5bTiWv1pvDPow+czb7N6kmFf1zQUxUs3ip4WPovBtQKmfpf9/i3DNkRILcoMLdZdKnn0aSaK66f0oxkSIc4nEkb3O9PbejVso6wLqSdDCh96d71gbHqOjyiZLBj2VlqalWvEPuo9GB0s2Uz2fxtFGMUQiZvH3jKR+9F4LwvKCc1K0E/+J4Np57JSfKgmid9QyL2r7nO19SVoVL3yBY7D8UxVIRw8sT/+JKXlnyh8roK7kaxDtW4+FMK6LT/QPvi8LkiNmA+eVv3kk9w==
    Hash of data:\nVPPxOVW2A38lCB810vuZbBH50KQaPSCouN0+tOpYDYs=
 

Implementing Basic Auth in Core WCF

WCF or Windows Communication Foundation was released initially in 2006 and was an important part of .NET Framework to create serverside services. It supports a lot of different protocols, not only HTTP(S), but also Net.Tcp, Msmq, Named pipes and more.

Sadly, .NET Core 1, when released in 2016, did not include WCF. The use of WCF has been more and more replaced by REST API over HTTP(S) using JWT tokens and not SAML.

But a community driven project supported by a multitude of companies including Microsoft and Amazon Web Services has been working on the Core WCF project and this project is starting to gain some more use, also allowing companies to migrate their platform services over to .NET.

I have looked at some basic stuff though, namely Basic Auth in Core WCF, and actually there is no working code sample for this. I have tapped into the ASP.NET Core pipeline to make it work by studying different code samples which has made part of it work, and I got it working. In this article I will explain how.

I use GenericIdentity to make it work. On the client side I have this extension method where I pass the username and password inside the soap envelope. I use .net6 client and service and service use CoreWCF version 1.5.1.

Source code for demo client is here: https://github.com/toreaurstadboss/CoreWCFWebClient1

The client is an ASP.NET Core MVC client who has added a Core WCF service as a connected service, generating a ServiceClient. The same type of service reference seen in .NET Framework in other words.

Client side setup for Core WCF Basic Auth

Source code for demo service is here: https://github.com/toreaurstadboss/CoreWCFService1

Extension method WithBasicAuth:
BasicHttpBindingClientFactory.cs

using System.ServiceModel;
using System.ServiceModel.Channels;

namespace CoreWCFWebClient1.Extensions
{
    public static class BasicHttpBindingClientFactory
    {

        /// <summary>
        /// Creates a basic auth client with credentials set in header Authorization formatted as 'Basic [base64encoded username:password]'
        /// Makes it easier to perform basic auth in Asp.NET Core for WCF
        /// </summary>
        /// <param name="username"></param>
        /// <param name="password"></param>
        /// <returns></returns>
        public static TServiceImplementation WithBasicAuth<TServiceContract, TServiceImplementation>(this TServiceImplementation client, string username, string password)
              where TServiceContract : class
                where TServiceImplementation : ClientBase<TServiceContract>, new()
        {
            string clientUrl = client.Endpoint.Address.Uri.ToString();

            var binding = new BasicHttpsBinding();
            binding.Security.Mode = BasicHttpsSecurityMode.Transport;
            binding.Security.Transport.ClientCredentialType = HttpClientCredentialType.Basic;

            string basicHeaderValue = "Basic " + Base64Encode($"{username}:{password}");
            var eab = new EndpointAddressBuilder(new EndpointAddress(clientUrl));
            eab.Headers.Add(AddressHeader.CreateAddressHeader("Authorization",  // Header Name
                string.Empty,           // Namespace
                basicHeaderValue));  // Header Value
            var endpointAddress = eab.ToEndpointAddress();

            var clientWithConfiguredBasicAuth = (TServiceImplementation) Activator.CreateInstance(typeof(TServiceImplementation), binding, endpointAddress)!;
            clientWithConfiguredBasicAuth.ClientCredentials.UserName.UserName = username;
            clientWithConfiguredBasicAuth.ClientCredentials.UserName.Password = username;

            return clientWithConfiguredBasicAuth;
        }

        private static string Base64Encode(string plainText)
        {
            var plainTextBytes = System.Text.Encoding.UTF8.GetBytes(plainText);
            return Convert.ToBase64String(plainTextBytes);
        }

    }
}

Example call inside a razor file in a .net6 web client, I made client and service from the WCF template :
Index.cshtml

@{

    string username = "someuser";
    string password = "somepassw0rd";

    var client = new ServiceClient().WithBasicAuth<IService, ServiceClient>(username, password);

    var result = await client.GetDataAsync(42);

    <h5>@Html.Raw(result)</h5>
}

I manage to set the identity via the call above, here is a screenshot showing this :

Setting up Basic Auth for serverside

Let's look at the serverside, it was created to start with as an ASP.NET Core .NET 6 with MVC Views solution. I added these Nugets to add CoreWCF, showing the entire .csproj since it also includes some important using statements :
CoreWCFService1.csproj

<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>true</ImplicitUsings>
  </PropertyGroup>
  <ItemGroup>
    <Using Include="CoreWCF" />
    <Using Include="CoreWCF.Configuration" />
    <Using Include="CoreWCF.Channels" />
    <Using Include="CoreWCF.Description" />
    <Using Include="System.Runtime.Serialization " />
    <Using Include="CoreWCFService1" />
    <Using Include="Microsoft.Extensions.DependencyInjection.Extensions" />
  </ItemGroup>
  <ItemGroup>
    <PackageReference Include="CoreWCF.Primitives" Version="1.5.1" />
    <PackageReference Include="CoreWCF.Http" Version="1.5.1" />
  </ItemGroup>
</Project>

Next up, in the file Program.cs different setup is added to add Basic Auth. In Program.cs , basic auth is set up in these code lines :
Program.cs

builder.Services.AddSingleton<IUserRepository, UserRepository>();

builder.Services.AddAuthentication("Basic").
            AddScheme<AuthenticationSchemeOptions, BasicAuthenticationHandler>
            ("Basic", null);
             

This adds authentication in services. We also make sure to add authentication itself after WebApplicationBuilder has been built, making sure also to set AllowSynchronousIO to true as usual. Below is listet the pipline setup of authentication, the StartsWithSegments should of course be adjusted in case you have multiple services:
Program.cs

app.Use(async (context, next) =>
{
    // Only check for basic auth when path is for the TransportWithMessageCredential endpoint only
    if (context.Request.Path.StartsWithSegments("/Service.svc"))
    {
        // Check if currently authenticated
        var authResult = await context.AuthenticateAsync("Basic");
        if (authResult.None)
        {
            // If the client hasn't authenticated, send a challenge to the client and complete request
            await context.ChallengeAsync("Basic");
            return;
        }
    }
    // Call the next delegate/middleware in the pipeline.
    // Either the request was authenticated of it's for a path which doesn't require basic auth
    await next(context);
});

We set up the servicemodel security like this to support transport mode security with the basic client credentials type.
Program.cs

 
app.UseServiceModel(serviceBuilder =>
{
    var basicHttpBinding = new BasicHttpBinding();
    basicHttpBinding.Security.Mode = BasicHttpSecurityMode.Transport;
    basicHttpBinding.Security.Transport.ClientCredentialType = HttpClientCredentialType.Basic;
    serviceBuilder.AddService<Service>(options =>
    {
        options.DebugBehavior.IncludeExceptionDetailInFaults = true;
    });
    serviceBuilder.AddServiceEndpoint<Service, IService>(basicHttpBinding, "/Service.svc");

    var serviceMetadataBehavior = app.Services.GetRequiredService<ServiceMetadataBehavior>();
    serviceMetadataBehavior.HttpsGetEnabled = true;
});
  

The BasicAuthenticationHandler looks like this:
BasicAuthenticationHandler.cs

	
using Microsoft.AspNetCore.Authentication;
using Microsoft.Extensions.Options;
using System.Security.Claims;
using System.Security.Principal;
using System.Text;
using System.Text.Encodings.Web;

public class BasicAuthenticationHandler : AuthenticationHandler<AuthenticationSchemeOptions>
{

    private readonly IUserRepository _userRepository;
    public BasicAuthenticationHandler(IOptionsMonitor<AuthenticationSchemeOptions> options,
        ILoggerFactory logger,
        UrlEncoder encoder,
        ISystemClock clock, IUserRepository userRepository) :
       base(options, logger, encoder, clock)
    {
        _userRepository = userRepository;
    }

    protected async override Task<AuthenticateResult> HandleAuthenticateAsync()
    {
        string? authTicketFromSoapEnvelope = await Request!.GetAuthenticationHeaderFromSoapEnvelope();

        if (authTicketFromSoapEnvelope != null && authTicketFromSoapEnvelope.StartsWith("basic", StringComparison.OrdinalIgnoreCase))
        {
            var token = authTicketFromSoapEnvelope.Substring("Basic ".Length).Trim();
            var credentialsAsEncodedString = Encoding.UTF8.GetString(Convert.FromBase64String(token));
            var credentials = credentialsAsEncodedString.Split(':');
            if (await _userRepository.Authenticate(credentials[0], credentials[1]))
            {
                var identity = new GenericIdentity(credentials[0]);
                var claimsPrincipal = new ClaimsPrincipal(identity);
                var ticket = new AuthenticationTicket(claimsPrincipal, Scheme.Name);
                return await Task.FromResult(AuthenticateResult.Success(ticket));
            }
        }

        return await Task.FromResult(AuthenticateResult.Fail("Invalid Authorization Header"));
    }

    protected override Task HandleChallengeAsync(AuthenticationProperties properties)
    {
        Response.StatusCode = 401;
        Response.Headers.Add("WWW-Authenticate", "Basic realm=\"thoushaltnotpass.com\"");
        Context.Response.WriteAsync("You are not logged in via Basic auth").Wait();
        return Task.CompletedTask;
    }

}

This authentication handler has got a flaw, if you enter the wrong password and username you get a 500 internal server error instead of the 401. I have not found out how to fix this yet.. Authenticate.Fail seems to short-circuit everything in case you enter wrong credentials. The _userRepository.Authenticate method is implemented as a dummy implementation, the user repo could for example do a database connection to look up the user via the provided credentials or some other means, maybe via ASP.NET Core MemberShipProvider ? The user repo looks like this:
(I)UserRepository.cs

  public interface IUserRepository
    {

        public Task<bool> Authenticate(string username, string password);
    }

    public class UserRepository : IUserRepository
    {
        public Task<bool> Authenticate(string username, string password)
        {
            //TODO: some dummie auth mechanism used here, make something more realistic such as DB user repo lookup or similar
            if (username == "someuser" && password == "somepassw0rd")
            {
                return Task.FromResult(true);
            }
            return Task.FromResult(false);
        }
    }
    
    
 

So I have implemented basic auth via reading out the credentials via Auth header inside soap envelope. I circumvent a lot of the Core WCF Auth by perhaps relying too much on the ASP.Net Core pipeline instead. Remember, WCF has to interop some with the ASP.NET Core pipeline to make it work properly and as long as we satisfy the demands of both the WCF and ASP.NET Core pipelines, we can make the authentication work. I managed to set the username via setting claims in the expected places of ServiceSecurityContext and CurrentPrincipal. The WCF service looks like this, note the use of the [Autorize] attribute :
Service.cs

   
public class Service : IService
 {

     [Authorize]
     public string GetData(int value)
     {
         return $"You entered: {value}. <br />The client logged in with transport security with BasicAuth with https (BasicHttpsBinding).<br /><br />The username is set inside ServiceSecurityContext.Current.PrimaryIdentity.Name: {ServiceSecurityContext.Current.PrimaryIdentity.Name}. <br /> This username is also stored inside Thread.CurrentPrincipal.Identity.Name: {Thread.CurrentPrincipal?.Identity?.Name}";
     }

     public CompositeType GetDataUsingDataContract(CompositeType composite)
     {
         if (composite == null)
         {
             throw new ArgumentNullException("composite");
         }
         if (composite.BoolValue)
         {
             composite.StringValue += "Suffix";
         }
         return composite;
     }
 }
   

I am mainly satisfied with this setup as it though is not optimal since ASP.NET Core don't seem to be able to work together with CoreWCF properly, instead we add the authentication as a soap envelope authorization header which we read out. I used some time to read out the authentication header, this is done on the serverside with the following extension method :
HttpRequestExtensions.cs

 
 
using System.IO.Pipelines;
using System.Text;
using System.Xml.Linq;

public static class HttpRequestExtensions
{

    public static async Task<string?> GetAuthenticationHeaderFromSoapEnvelope(this HttpRequest request)
    {
        ReadResult requestBodyInBytes = await request.BodyReader.ReadAsync();
        string body = Encoding.UTF8.GetString(requestBodyInBytes.Buffer.FirstSpan);
        request.BodyReader.AdvanceTo(requestBodyInBytes.Buffer.Start, requestBodyInBytes.Buffer.End);

        string authTicketFromHeader = null;

        if (body?.Contains(@"http://schemas.xmlsoap.org/soap/envelope/") == true)
        {
            XNamespace ns = "http://schemas.xmlsoap.org/soap/envelope/";
            var soapEnvelope = XDocument.Parse(body);
            var headers = soapEnvelope.Descendants(ns + "Header").ToList();

            foreach (var header in headers)
            {
                var authorizationElement = header.Element("Authorization");
                if (!string.IsNullOrWhiteSpace(authorizationElement?.Value))
                {
                    authTicketFromHeader = authorizationElement.Value;
                    break;
                }
            }
        }

        return authTicketFromHeader;
    }

} 
 
 

Note the use of BodyReader and method AdvanceTo. This was the only way to rewind the Request stream after reading the HTTP soap envelope header for Authorization, it took me hours to figure out why this failed in ASP.NET Core pipeline, until I found some tips in a Github discussion thread on Core WCF mentioning the error and a suggestion in a comment there. See more documentation about BodyWriter and BodyReader here from MVP Steve Gordon here: https://www.stevejgordon.co.uk/using-the-bodyreader-and-bodywriter-in-asp-net-core-3-0

Increasing timeout in CoreWCF project for client

I have tested out CoreWCF a bit and it is good to see WCF once again in a modern framework such as ASP.NET Core. Here is how you can increase timeouts in CoreWCF. You can put the timeout into an appsettings file too if you want. First off, after having added a Service Reference to your WCF service. Look inside the Reference.cs file. Make note of:

1. Namespace in the Reference.cs file
2. Class name of the client

My client uses these Nuget packages in its csproj :

  
  
  <ItemGroup>
    <PackageReference Include="System.ServiceModel.Duplex" Version="4.10.*" />
    <PackageReference Include="System.ServiceModel.Federation" Version="4.10.*" />
    <PackageReference Include="System.ServiceModel.Http" Version="4.10.*" />
    <PackageReference Include="System.ServiceModel.NetTcp" Version="4.10.*" />
    <PackageReference Include="System.ServiceModel.Security" Version="4.10.*" />
  </ItemGroup>
  
    

 
 <ItemGroup>
    <PackageReference Include="CoreWCF.Primitives" Version="1.*" />
    <PackageReference Include="CoreWCF.Http" Version="1.*" />
  </ItemGroup> 
 

Look inside the Reference.cs file, a method called ConfigureEndpoint is listed :

	
    
    [System.Diagnostics.DebuggerStepThroughAttribute()]
    [System.CodeDom.Compiler.GeneratedCodeAttribute("Microsoft.Tools.ServiceModel.Svcutil", "2.1.0")]
    public partial class ServiceClient : System.ServiceModel.ClientBase, MyService.IService
    {
        
        /// 

        /// Implement this partial method to configure the service endpoint.
        /// 
        /// The endpoint to configure
        /// The client credentials
        static partial void ConfigureEndpoint(System.ServiceModel.Description.ServiceEndpoint serviceEndpoint, System.ServiceModel.Description.ClientCredentials clientCredentials);

        //more code 
    
    

Next up, implementing this method to configured the binding.

	
    
namespace MyService
{
    public partial class ServiceClient
    {

        /// <summary>
        /// Implement this partial method to configure the service endpoint.
        /// </summary>
        /// <param name="serviceEndpoint">The endpoint to configure</param>
        /// <param name="clientCredentials">The client credentials</param>
        static partial void ConfigureEndpoint(System.ServiceModel.Description.ServiceEndpoint serviceEndpoint, System.ServiceModel.Description.ClientCredentials clientCredentials)
        {
            serviceEndpoint.Binding.OpenTimeout 
                = serviceEndpoint.Binding.CloseTimeout
                = serviceEndpoint.Binding.ReceiveTimeout
                = serviceEndpoint.Binding.SendTimeout = TimeSpan.FromSeconds(15);
        }

    }
}
    
    

We also want to be able to configure the timeout here. Lets add the following nuget packages also to the client (I got a .NET 6 console app):

	
 <PackageReference Include="Microsoft.Extensions.Configuration" Version="6.0.0" />
 <PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="6.0.0" />
    

We can also avoid hardcoding timeouts by adding appsettings.json to our project and set the file to copy to output folder. If you are inside a console project you can add json config file like this, preferably registering it in some shared setup in Program.cs, but I found it a bit challenging to consume it from a static method I ended up with this :

 

            /// 

            /// Implement this partial method to configure the service endpoint.
            /// 
            /// The endpoint to configure
            /// The client credentials
            static partial void ConfigureEndpoint(System.ServiceModel.Description.ServiceEndpoint serviceEndpoint, System.ServiceModel.Description.ClientCredentials clientCredentials)
            {
                var serviceProvider = new ServiceCollection()
                    .AddSingleton(_ =>
                        new ConfigurationBuilder()
                            .SetBasePath(Path.Combine(AppContext.BaseDirectory))
                            .AddJsonFile("appsettings.json", optional: true)
                            .Build())
                    .BuildServiceProvider();
    
                var config = serviceProvider.GetService();
    
                int timeoutInSeconds = int.Parse(config!["ServiceTimeoutInSeconds"]);
                serviceEndpoint.Binding.OpenTimeout
                    = serviceEndpoint.Binding.CloseTimeout
                    = serviceEndpoint.Binding.ReceiveTimeout
                    = serviceEndpoint.Binding.SendTimeout = TimeSpan.FromSeconds(timeoutInSeconds);
            }
                 
               

And we have our appsettings.json file :

    {
      "ServiceTimeoutInSeconds" :  9
    }
    

The CoreWCF project got an upgrade tool that will do a lot of the migration for you. WCF had a lot of config settings and having an appsettings.json file for every setting will be some work. The upgrade tool should take care of generating some of these config values and add them into dedicated json files for this.