Thursday 16 December 2021

AngularJs directive for clearing a text field

I wrote an AngularJs directive at work today for clearing a text field. We still use this in multiple projects for front-end (although I have worked more with Angular than AngularJs last 2-3 years). The directive ended up like this (we use Bootstrap v2.3.2) :

angular.module('formModule').directive('addClearTextFieldBtn', function ($compile) {
        function link(scope, element, attrs) {
            var targetId = attrs.id;
            var targetNgModel = attrs.ngModel;
            var minimumChars = attrs.clearTextFieldBtnMintextlength ? attrs.clearTextFieldBtnMintextlength : "1";
            var emptyValue = "''";
            var templateAppend = '<i id="clear' + targetId + '" ng-if="' + targetNgModel + ' && + ' + targetNgModel + '.length >= ' + minimumChars + '"' + 'ng-click="' + targetNgModel;
            templateAppend += ' = ' + emptyValue + '" class="glyphicon icon-remove form-control-feedback" title="Tøm innhold" style="cursor:pointer; pointer-events: all;" tooltip="clear"></i >';
            var clearButton = angular.element(templateAppend);
            clearButton.insertAfter(element);
            $compile(clearButton)(scope);
        }
        return {
            restrict: 'A',
            replace: false,
            link: link
        };
    });

Example usage inside a HTML helper in MVC for example:
add_clear_text_field_btn = "model.Icd10", data_clear_text_field_btn_mintextlength="3"
We pass in a HTML5 data value attribute to specify the minimum length to show the button to clear the field.

Sunday 12 December 2021

Displaying errors in Event Log with Out-GridView in Powershell

A user friendly way to view errors in Event log source from Powershell.
$rawUI = $Host.UI.RawUI
$oldSize = $rawUI.BufferSize
$typeName = $oldSize.GetType( ).FullName
$newSize = New-Object $typeName (500, $oldSize.Height)
$rawUI.BufferSize = $newSize

get-eventlog -logname someacmecompanyname | where-object { $_.source -like '*someeventlogsourcename*' -and $_.EntryType -in ('Error', 'Warning', 'Critical') } | out-gridview

Wednesday 24 November 2021

Scanning solutions for NUnit test adapter via Powershell

Checking that we have added test adapter for NUnit so that our tests in Azure Devops are run

A challenge with running tests inside Powershell can be if NUnit test adapter Nuget package is missing from the solution. If you run test using NUnit 2.x, you require NUnitTestAdapter. If you use NUnit 3.x, NUnit3TestAdapter is required. The following Powershell script can be used to check if we have added a Nuget package reference at least to one such test project in the solution. We have here some tests that will list up all PackageReference in csproj files of the solution. Note: this requires the following setup of your Nuget package references listed in the solution.
  • You have to have csproj projects in the solution
  • You must use PackageReference, i.e. list up nuget packages in the csproj file. This will not work if you instead use packages lock json format or packages.config.
The Powershell functions are these:
 
 
 
 Function Get-ProjectInSolution {
    [CmdletBinding()] param (
        [Parameter()][string]$Solution
    )
    $SolutionPath = $Solution
    $SolutionFile = Get-Item $SolutionPath
    $SolutionFolder = $SolutionFile.Directory.FullName

    Get-Content $Solution |
        Select-String 'Project\(' |
        ForEach-Object {
            $projectParts = $_ -Split '[,=]' | ForEach-Object { $_.Trim('[ "{}]') }
            [PSCustomObject]@{
                File = $projectParts[2]
                Guid = $projectParts[3]
                Name = $projectParts[1]
            }
        } |
        Where-Object File -match "csproj$" |
        ForEach-Object {
            Add-Member -InputObject $_ -NotePropertyName FullName -NotePropertyValue (Join-Path $SolutionFolder $_.File) -PassThru
        }
}

Function Get-TestProjectInSolution {
[CmdletBinding()] param (
[Parameter()][string]$Solution)
  
  $projects = & Get-ProjectInSolution $Solution
  $testProjects = $projects | Where-Object { $_.Name -like '*Test*' }
  return $testProjects
}


Function Get-PackagesInProject {
[CmdletBinding()] param (
[Parameter()][string]$ProjectFile)

Get-Content $ProjectFile | Write-Host 
}


# Get-ProjectInSolution "C:\dev\somesolution\someacme.sln" 

Function List-PackagesOfTestProjectInSolution {
[CmdletBinding()] param (
[Parameter()][string]$SolutionFile)

  & Get-TestProjectInSolution $SolutionFile | ForEach-Object {
   $filePath = $_.FullName 
   Write-Host $filePath
  (Get-Content $_.FullName | Find "<PackageReference Include")
}
}

 
    Function Get-PackagesOfTestProjectInSolution {
[CmdletBinding()] param (
[Parameter()][string]$SolutionFile)

$dict = @{}

  & Get-TestProjectInSolution $SolutionFile | ForEach-Object {
    $filePath = $_.FullName 
    # Write-Host $filePath
    if (-not $dict.ContainsKey($filePath)) {
        $dict[$filePath] = (Get-Content $_.FullName | Find "<PackageReference Include")
    }
    return $dict 
  }

}

Function Has-NunitTestAdapterPackageInTestProjectinSolution {
[CmdletBinding()] param (
[Parameter()][string]$SolutionFile) 
 $packagesDict = Get-PackagesOfTestProjectInSolution $SolutionFile
$allPackagesString = $packagesDict.Values
$isNunitTestAdapterFound = ($allPackagesString -like "*NUnit*TestAdapter*").Length -gt 0
return $isNunitTestAdapterFound
}


Get-PackagesOfTestProjectInSolution "C:\dev\someacme\someacme.sln" 

$isNunitTestAdapterPresent = Has-NunitTestAdapterPackageInTestProjectinSolution "C:\dev\someacme\somecme.sln" 

Write-Host "Is NUnit test adapter added?" $isNunitTestAdapterPresent

    
    
    
    
For example, we could run the function call : List-PackagesOfTestProjectInSolution "C:\dev\someacme\someacme.sln" And we get our lists of package references in that solution (here we only look inside projects with a name containing "Test":

Friday 9 July 2021

Immutable lists in C# - Adding a wrapper class

This article will discuss the immutable collections in C#, more precisely immutable lists of generic type T wrapped inside a class. This makes it possible to easier use immutable lists and these lists can only be altered via functional calls. Remember that an immutable list always returns a new immutable list. For easier use, we can have a wrapper for this. First of, inside Linqpad 5, being used in this article, hit F4. In case you want to use Visual Studio instead, the same code should work there (except Linqpad's Dump method). In the tab Additional referencesNow choose Add Nuget.. Then seach for System.Collections.Immutable. After selecting this Nuget package, choose the tab Additional Namesapce Imports. Now paste this demo code:
 
 void Main()
{
	var numbersInImmutableList = new ImmutableWrappedList<int>();
	numbersInImmutableList.AddRange(new[] { 3, 1, 4, 1, 5, 9, 2 }); 
	numbersInImmutableList.AddRange(new[]{ 2, 7, 1, 8, 2, 1, 8 });
	numbersInImmutableList.RemoveAt(2);
	numbersInImmutableList.Contents.Dump(); 	
}

public class ImmutableWrappedList<T>  {
	public ImmutableList<T> _internalList;
    	public ImmutableList<T> Contents => _internalList;
	
	public ImmutableWrappedList()
	{
		_internalList = ImmutableList.Create<T>(); 	
	}
	
	public void Clear() => _internalList.Clear(); 	
	public void AddRange(IEnumerable<T> itemsToAdd) => _internalList = _internalList.AddRange(itemsToAdd);
	public void Add(T itemToAdd) => _internalList = _internalList.Add(itemToAdd);
	public void Remove(T itemToAdd) => _internalList = _internalList.Remove(itemToAdd);
	public void RemoveAt(int index) => _internalList = _internalList.RemoveAt(index);
	public void Insert(T itemToAdd, int position) => _internalList = _internalList.Insert(position, itemToAdd);
}


 
As we can see, the wrapper class can add items to the immutable collections and we also reassign the result modifying operation to the same _internalList field, which has a private setter and is initialized to an empty array in the constructor. This gives you mutability to the immutable collection without having to remember to reassign the variable, which is error prone in itself. Note - we have called the _internalList and you see that we can get thi What is the benefit of this ? Well, although we can reach into the internal collection with the Contents method here, the immutable list is still immutable. If you want to change it, you have to call specific methods here on it offered in the wrapping class. So, data-integrity wise, we have data that only can change via the methods offered in the wrapping class. A collection which is not immutable can be changed in many ways only by giving access to it. We still have control over the data via the wrapper and we make it easier to consume the immutable class by reassigning the collection.

Wednesday 7 July 2021

Dapper - Inner Joins between two tables - Helper methods

Many developers use Entity Framework (EF) today as the library of their data access library to communicate against the database. EF is a ORM, object-relational mapper and while it boasts much functionality like change tracking and mapping relationships, Dapper at the other line of ORMs is a Micro-ORM. A Micro-ORM has less functionality, but offers usually more speed and less overhead. Dapper is a great Micro-ORM, however, writing SQL manually is often error-prone or tedious. Some purists love writing the SQL manually and be sure which SQL they send off to the DB. That is much of the point of Dapper. However, lending a hand to developers in building their SQL should still be allowed. The query compilation time added to such helper methods are miniscule anyways compared to the heavy overhead of an advanced ORM like EF. Anyways, the code in this article shows some code I am working with for building inner joins between to tables. The relationship between the two tables are 1:1 in my test case and the inner join does for now not support a where predicate filter, although adding such a filter should be easy. The source code for DapperUtils of mine is available on GitHub: https://github.com/toreaurstadboss/DapperUtils
First, we make use of SqlBuilder from DapperUtils addon lib for Dapper.

using Dapper;
using System.Collections.Generic;
using System.Linq;
using System.Text.RegularExpressions;

namespace DapperUtils.ToreAurstadIT
{
    /// <summary>
    /// Original is fetched from: https://raw.githubusercontent.com/DapperLib/Dapper/main/Dapper.SqlBuilder/SqlBuilder.cs
    /// 
    /// </summary>
    public class SqlBuilder
    {
        private readonly Dictionary<string, Clauses> _data = new Dictionary<string, Clauses>();
        private int _seq;

        private class Clause
        {
            public string Sql { get; set; }
            public object Parameters { get; set; }
            public bool IsInclusive { get; set; }
        }

        private class Clauses : List<Clause>
        {
            private readonly string _joiner, _prefix, _postfix;

            public Clauses(string joiner, string prefix = "", string postfix = "")
            {
                _joiner = joiner;
                _prefix = prefix;
                _postfix = postfix;
            }

            public string ResolveClauses(DynamicParameters p)
            {
                foreach (var item in this)
                {
                    p.AddDynamicParams(item.Parameters);
                }
                return this.Any(a => a.IsInclusive)
                    ? _prefix +
                      string.Join(_joiner,
                          this.Where(a => !a.IsInclusive)
                              .Select(c => c.Sql)
                              .Union(new[]
                              {
                                  " ( " +
                                  string.Join(" OR ", this.Where(a => a.IsInclusive).Select(c => c.Sql).ToArray()) +
                                  " ) "
                              }).ToArray()) + _postfix
                    : _prefix + string.Join(_joiner, this.Select(c => c.Sql).ToArray()) + _postfix;
            }
        }

        public class Template
        {
            private readonly string _sql;
            private readonly SqlBuilder _builder;
            private readonly object _initParams;
            private int _dataSeq = -1; // Unresolved

            public Template(SqlBuilder builder, string sql, dynamic parameters)
            {
                _initParams = parameters;
                _sql = sql;
                _builder = builder;
            }

            private static readonly Regex _regex = new Regex(@"\/\*\*.+?\*\*\/", RegexOptions.Compiled | RegexOptions.Multiline);

            private void ResolveSql()
            {
                if (_dataSeq != _builder._seq)
                {
                    var p = new DynamicParameters(_initParams);

                    rawSql = _sql;

                    foreach (var pair in _builder._data)
                    {
                        rawSql = rawSql.Replace("/**" + pair.Key + "**/", pair.Value.ResolveClauses(p));
                    }
                    parameters = p;

                    // replace all that is left with empty
                    rawSql = _regex.Replace(rawSql, "");

                    _dataSeq = _builder._seq;
                }
            }

            private string rawSql;
            private object parameters;

            public string RawSql
            {
                get { ResolveSql(); return rawSql; }
            }

            public object Parameters
            {
                get { ResolveSql(); return parameters; }
            }
        }

        public Template AddTemplate(string sql, dynamic parameters = null) =>
            new Template(this, sql, parameters);

        protected SqlBuilder AddClause(string name, string sql, object parameters, string joiner, string prefix = "", string postfix = "", bool isInclusive = false)
        {
            if (!_data.TryGetValue(name, out Clauses clauses))
            {
                clauses = new Clauses(joiner, prefix, postfix);
                _data[name] = clauses;
            }
            clauses.Add(new Clause { Sql = sql, Parameters = parameters, IsInclusive = isInclusive });
            _seq++;
            return this;
        }

        public SqlBuilder Intersect(string sql, dynamic parameters = null) =>
            AddClause("intersect", sql, parameters, "\nINTERSECT\n ", "\n ", "\n", false);

        public SqlBuilder InnerJoin(string sql, dynamic parameters = null) =>
            AddClause("innerjoin", sql, parameters, "\nINNER JOIN ", "\nINNER JOIN ", "\n", false);

        public SqlBuilder LeftJoin(string sql, dynamic parameters = null) =>
            AddClause("leftjoin", sql, parameters, "\nLEFT JOIN ", "\nLEFT JOIN ", "\n", false);

        public SqlBuilder RightJoin(string sql, dynamic parameters = null) =>
            AddClause("rightjoin", sql, parameters, "\nRIGHT JOIN ", "\nRIGHT JOIN ", "\n", false);

        public SqlBuilder Where(string sql, dynamic parameters = null) =>
            AddClause("where", sql, parameters, " AND ", "WHERE ", "\n", false);

        public SqlBuilder OrWhere(string sql, dynamic parameters = null) =>
            AddClause("where", sql, parameters, " OR ", "WHERE ", "\n", true);

        public SqlBuilder OrderBy(string sql, dynamic parameters = null) =>
            AddClause("orderby", sql, parameters, " , ", "ORDER BY ", "\n", false);

        public SqlBuilder Select(string sql, dynamic parameters = null) =>
            AddClause("select", sql, parameters, " , ", "", "\n", false);

        public SqlBuilder AddParameters(dynamic parameters) =>
            AddClause("--parameters", "", parameters, "", "", "", false);

        public SqlBuilder Join(string sql, dynamic parameters = null) =>
            AddClause("join", sql, parameters, "\nJOIN ", "\nJOIN ", "\n", false);

        public SqlBuilder GroupBy(string sql, dynamic parameters = null) =>
            AddClause("groupby", sql, parameters, " , ", "\nGROUP BY ", "\n", false);

        public SqlBuilder Having(string sql, dynamic parameters = null) =>
            AddClause("having", sql, parameters, "\nAND ", "HAVING ", "\n", false);

        public SqlBuilder Set(string sql, dynamic parameters = null) =>
             AddClause("set", sql, parameters, " , ", "SET ", "\n", false);

    }
}

Using SqlBuilder, we can define a Sql template and add extension methods and helper methods required to build and retrieve the inner join. The helper methods in use are added also below the extension method InnerJoin. Make note that we use SqlBuilder here to do much of the SQL template processing to end up
with the SQL that is sent to the DB (RawSql property of SqlBuilder instance).

        /// <summary>
        /// Inner joins the left and right tables by specified left and right key expression lambdas.
        /// This uses a template builder and a shortcut to join two tables without having to specify any SQL manually
        /// and gives you the entire inner join result set. It is an implicit requirement that the <paramref name="leftKey"/>
        /// and <paramref name="rightKey"/> are compatible data types as they are used for the join.
        /// This method do for now not allow specifying any filtering (where-clause) or logic around the joining besides
        /// just specifying the two columns to join.
        /// </summary>
        /// <typeparam name="TLeftTable">Type of left table</typeparam>
        /// <typeparam name="TRightTable">Type of right table</typeparam>
        /// <param name="connection">IDbConnection to the DB</param>
        /// <param name="leftKey">Member expression of the left table in the join</param>
        /// <param name="rightKey">Member expression to the right table in the join</param>
        /// <returns>IEnumerable of ExpandoObject. Tip: Iterate through the IEnumerable and save each ExpandoObject into a variable of type dynamic to access the variables more conveniently if desired.</returns>
        public static IEnumerable<ExpandoObject> InnerJoin<TLeftTable, TRightTable>(this IDbConnection connection, 
            Expression<Func<TLeftTable, object>> leftKey, Expression<Func<TRightTable, object>> rightKey)
        {
            var builder = new SqlBuilder();
            string leftTableSelectClause = string.Join(",", GetPublicPropertyNames<TLeftTable>("l"));
            string rightTableSelectClause = string.Join(",", GetPublicPropertyNames<TRightTable>("r"));
            string leftKeyName = GetMemberName(leftKey);
            string rightKeyName = GetMemberName(rightKey); 
            string leftTableName = GetDbTableName<TLeftTable>();
            string rightTableName = GetDbTableName<TRightTable>(); 
            string joinSelectClause = $"select {leftTableSelectClause}, {rightTableSelectClause} from {leftTableName} l /**innerjoin**/";
            var selector = builder.AddTemplate(joinSelectClause);
            builder.InnerJoin($"{rightTableName} r on l.{leftKeyName} = r.{rightKeyName}");
            var joinedResults = connection.Query(selector.RawSql, selector.Parameters)
                .Select(x => (ExpandoObject)DapperUtilsExtensions.ToExpandoObject(x)).ToList();
            return joinedResults;
        }
        
          private static string[] GetPublicPropertyNames<T>(string tableQualifierPrefix = null) {
            return typeof(T).GetProperties(System.Reflection.BindingFlags.Public | System.Reflection.BindingFlags.Instance)
                 .Where(x => !IsNotMapped(x))
                 .Select(x => !string.IsNullOrEmpty(tableQualifierPrefix) ? tableQualifierPrefix + "." + x.Name : x.Name).ToArray();
        }

     private static bool IsNotMapped(PropertyInfo x)
        {
            var notmappedAttr = x.GetCustomAttributes<NotMappedAttribute>()?.OfType<NotMappedAttribute>().FirstOrDefault();
            return notmappedAttr != null;
        }
       /// <summary>
        /// Returns database table name, either via the System.ComponentModel.DataAnnotations.Schema.Table attribute
        /// if it exists, or just the name of the <typeparamref name="TClass"/> type parameter. 
        /// </summary>
        /// <typeparam name="TClass"></typeparam>
        /// <returns></returns>
        private static string GetDbTableName<TClass>()
        {
            var tableAttribute = typeof(TClass).GetCustomAttributes(typeof(TableAttribute), false)?.FirstOrDefault() as TableAttribute;
            if (tableAttribute != null)
            {
                if (!string.IsNullOrEmpty(tableAttribute.Schema))
                {
                    return $"[{tableAttribute.Schema}].[{tableAttribute.Name}]";
                }
                return tableAttribute.Name;
            }
            return typeof(TClass).Name;
        }     

        private static string GetMemberName<T>(Expression<Func<T, object>> expression)
        {
            switch (expression.Body)
            {
                case MemberExpression m:
                    return m.Member.Name;
                case UnaryExpression u when u.Operand is MemberExpression m:
                    return m.Member.Name;
                default:
                    throw new NotImplementedException(expression.GetType().ToString());
            }
        }

        /// <summary>
        /// Returns database table name, either via the System.ComponentModel.DataAnnotations.Schema.Table attribute
        /// if it exists, or just the name of the <typeparamref name="TClass"/> type parameter. 
        /// </summary>
        /// <typeparam name="TClass"></typeparam>
        /// <returns></returns>
        private static string GetDbTableName<TClass>()
        {
            var tableAttribute = typeof(TClass).GetCustomAttributes(typeof(TableAttribute), false)?.FirstOrDefault() as TableAttribute;
            if (tableAttribute != null)
            {
                if (!string.IsNullOrEmpty(tableAttribute.Schema))
                {
                    return $"[{tableAttribute.Schema}].[{tableAttribute.Name}]";
                }
                return tableAttribute.Name;
            }
            return typeof(TClass).Name;
        }     

        public static ExpandoObject ToExpandoObject(object value)
        {
            IDictionary<string, object> dapperRowProperties = value as IDictionary<string, object>;
            IDictionary<string, object> expando = new ExpandoObject();
            if (dapperRowProperties == null)
            {
                return expando as ExpandoObject;
            }
            foreach (KeyValuePair<string, object> property in dapperRowProperties)
            {
                if (!expando.ContainsKey(property.Key))
                {
                    expando.Add(property.Key, property.Value);
                }
                else
                {
                    //prefix the colliding key with a random guid suffixed 
                    expando.Add(property.Key + Guid.NewGuid().ToString("N"), property.Value);
                } 
            }
            return expando as ExpandoObject;
        }       
        
        

Here are some Nuget packages in use in the small lib functions here are in test project too:

	   <!-- lib project .NET 5 -->
       <PackageReference Include="Dapper" Version="2.0.90" />
	   <PackageReference Include="Microsoft.CSharp" Version="4.7.0" />
	   <PackageReference Include="System.ComponentModel.Annotations" Version="5.0.0" />
       
       <!-- test project-->
        <PackageReference Include="FluentAssertions" Version="5.10.3" />
		<PackageReference Include="Microsoft.CSharp" Version="4.7.0" />
		<PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="3.1.16" />
		<PackageReference Include="Microsoft.Extensions.Options.ConfigurationExtensions" Version="3.1.16" />
		<PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.10.0" />
		<PackageReference Include="Newtonsoft.Json" Version="13.0.1" />
		<PackageReference Include="NUnit" Version="3.13.2" />
		<PackageReference Include="NUnit3TestAdapter" Version="4.0.0" />
		<PackageReference Include="System.ComponentModel.Annotations" Version="5.0.0" />
		<PackageReference Include="System.Data.SqlClient" Version="4.8.2" />

Two unit tests shows how easier syntax we get with this helper method. The downside is that you cant fully control the sql yourself, but the benefit is quicker to implement.
  
       [Test]
        public void InnerJoinWithManualSqlReturnsExpected()
        {
            var builder = new SqlBuilder();
            var selector = builder.AddTemplate("select p.ProductID, p.ProductName, p.CategoryID, c.CategoryName, s.SupplierID, s.City from products p /**innerjoin**/");
            builder.InnerJoin("categories c on c.CategoryID = p.CategoryID");
            builder.InnerJoin("suppliers s on p.SupplierID = s.SupplierID");
            dynamic joinedproductsandcategoryandsuppliers = Connection.Query(selector.RawSql, selector.Parameters).Select(x => (ExpandoObject)DapperUtilsExtensions.ToExpandoObject(x)).ToList();
            var firstRow = joinedproductsandcategoryandsuppliers[0];
            Assert.AreEqual(firstRow.ProductID + firstRow.ProductName + firstRow.CategoryID + firstRow.CategoryName + firstRow.SupplierID + firstRow.City, "1Chai1Beverages1London");
        }

        [Test]
        public void InnerJoinWithoutManualSqlReturnsExpected()
        {
            var joinedproductsandcategory = Connection.InnerJoin<Product, Category>(l => l.CategoryID, r => r.CategoryID);
            dynamic firstRow = joinedproductsandcategory.ElementAt(0);
            Assert.AreEqual(firstRow.ProductID + firstRow.ProductName + firstRow.CategoryID + firstRow.CategoryName + firstRow.SupplierID, "1Chai1Beverages1");
        }
  
Our POCO classes used in the tests are these two. We use the Nuget package System.ComponentModel.Annotations and attributes TableName and NotMapped to control the SQL built here to specify the DB table name for the POCO (if they are the same, the name of the type is used as fallback if attribute TableName is missing) and NotMapped in case there are properties like relationship properties ("navigation properties in EF for Dapper") that should not be used in the SQL select clause.
 
 using System.ComponentModel.DataAnnotations.Schema;

namespace DapperUtils.ToreAurstadIT.Tests
{
    [Table("Products")]
    public class Product
    {
        public int ProductID { get; set; }
        public string ProductName { get; set; }
        public int? SupplierID { get; set; }
        public int? CategoryID { get; set; }
        public string QuantityPerUnit { get; set; }
        public decimal? UnitPrice { get; set; }
        public short? UnitsInStock { get; set; }
        public short? UnitsOnOrder { get; set; }
        public short? ReorderLevel { get; set; }
        public bool? Discontinued { get; set; }
        [NotMapped]
        public Category Category { get; set; }
    }
}

using System.ComponentModel.DataAnnotations.Schema;

namespace DapperUtils.ToreAurstadIT.Tests
{
    [Table("Categories")]
    public class Category
    {
        public int CategoryID { get; set; }
        public string CategoryName { get; set; }
        public string Description { get; set; }
        public byte Picture { get; set; }
    }
}

 
In the end, we have a easy way to do a standard join. An improvement here could be the following:
  • Support for where predicates to filter the joins
  • More control on the join condition if desired
  • Support for joins accross three tables (or more?) - SqlBuilder already supports this, what is missing is lambda expression support for Intellisense support
  • What if a property does not match against db column ? Should support ColumnName attribute from System.ComponentModel.DataAnnotations.
  • Investigate other join types such as left outer joins - this should be just a minor adjustment actually.

Thursday 1 July 2021

SelectMany / Flattening multiple arrays at arbitrary depth in Typescript (Javascript)

I just added a flatten method of my SimpleTsLinq library today! The Github repo is at: The Npm page is at: This method can flatten multiple arrays at desired depth (defaults to Infinity) and each array itself may have arbitrary depth. The end result is that the multiple (nested arrays) are returned as a flat, single array. Much similar to SelectMany in Linq! First I added the method to generic interface Array below
 
 export { } //creating a module of below code
declare global {
  type predicate<T> = (arg: T) => boolean;
  type sortingValue<T> = (arg: T) => any;
  type keySelector<T> = (arg: T) => any;
  type resultSelector<T, TInner> = (arg: T, arg2: TInner) => any;
  interface Array<T> {
    AddRange<T>(itemsToAdd: T[]);
    InsertRange<T>(index: number, itemsToAdd: T[]);
    RemoveAt(index: number): T;
    RemoveWhere<T>(condition: predicate<T>): T[];
    FirstOrDefault<T>(condition: predicate<T>): T;
    SingleOrDefault<T>(condition: predicate<T>): T;
    First<T>(condition: predicate<T>): T;
    Single<T>(condition: predicate<T>): T;
    LastOrDefault<T>(condition: predicate<T>): T;
    Join<T, TInner>(otherArray: TInner[], outerKeySelector: keySelector<T>,
      innerKeySelector: keySelector<TInner>, res: resultSelector<T, TInner>): any[];
    Where<T>(condition: predicate<T>): T[];
    Count<T>(): number;
    CountBy<T>(condition: predicate<T>): number;
    Select<T>(...properties: (keyof T)[]): any[];
    GroupBy<T>(groupFunc: (arg: T) => string): any[];
    EnumerableRange(start: number, count: number): number[];
    Any<T>(condition: predicate<T>): boolean;
    Contains<T>(item: T): boolean;
    All<T>(condition: predicate<T>): boolean;
    MaxSelect<T>(property: (keyof T)): any;
    MinSelect<T>(property: (keyof T)): any;
    Average<T>(): number;
    AverageSelect<T>(property: (keyof T)): number;
    Max(): any;
    Min(): any;
    Sum(): any;
    Reverse<T>(): T[];
    Empty<T>(): T[];
    Except<T>(otherArray: T[]): T[];
    Intersect<T>(otherArray: T[]): T[];
    Union<T>(otherArray: T[]): T[];
    Cast<TOtherType>(TOtherType: Function): TOtherType[];
    TryCast<TOtherType>(TOtherType: Function): TOtherType[];
    GetProperties<T>(TClass: Function, sortProps: boolean): string[];
    Concat<T>(otherArray: T[]): T[];
    Distinct<T>(): T[];
    DistinctBy<T>(property: (keyof T)): any;
    SumSelect<T>(property: (keyof T)): any;
    Intersect<T>(otherArray: T[]): T[];
    IntersectSelect<T>(property: (keyof T), otherArray: T[]): T[];
    MinSelect<T>(property: (keyof T)): any;
    OrderBy<T>(sortMember: sortingValue<T>): T[];
    OrderByDescending<T>(sortMember: sortingValue<T>): T[];
    ThenBy<T>(sortMember: sortingValue<T>): T[];
    OfType<T>(compareObject: T): T[];
    SequenceEqual<T>(compareArray: T): boolean;
    Take<T>(count: number): T[];
    ToDictionary<T>(keySelector: (arg: T) => any): any;
    TakeWhile<T>(condition: predicate<T>): T[];
    SkipWhile<T>(condition: predicate<T>): T[];
    Skip<T>(count: number): T[];
    defaultComparerSort<T>(x: T, y: T);
    ElementAt<T>(index: number);
    ElementAtOrDefault<T>(index: number);
    Aggregate<T>(accumulator: any, currentValue: any, reducerFunc: (accumulator: any, currentValue: any) => any): any;
    AggregateSelect<T>(property: (keyof T), accumulator: any, currentValue: any, reducerFunc: (accumulator: any, currentValue: any) => any): any;
    Flatten<T>(otherArrays: T[][], depth: number): T[];
  }
}
 

Now we can implement the method as follows:
 
  
if (!Array.prototype.Flatten) {
  Array.prototype.Flatten = function <T>(otherArrays: T[][] = null, depth = Infinity) {
    let flattenedArrayOfThis = [...flatten(this, depth)];
    if (otherArrays == null || otherArrays == undefined) {
      return flattenedArrayOfThis;
    }
    return [...flattenedArrayOfThis, ...flatten(otherArrays, depth)];
  }
}

function* flatten(array, depth) {
  if (depth === undefined) {
    depth = 1;
  }
  for (const item of array) {
    if (Array.isArray(item) && depth > 0) {
      yield* flatten(item, depth - 1);
    } else {
      yield item;
    }
  }
}

 
The implementation uses a generator (identified by the * suffix) method which is recursively called if we have an array within an array Two tests below are run in Karma to test it out.
 
    it('can flatten multiple arrays into a single array', () => {
    let oneArray = [1, 2, [3, 3]];
    let anotherArray = [4, [4, 5], 6];
    let thirdArray = [7, 7, [7, 7]];
    let threeArrays = [oneArray, anotherArray, thirdArray];
    let flattenedArrays = oneArray.Flatten([anotherArray, thirdArray], Infinity);
    let isEqualInContentToExpectedFlattenedArray = flattenedArrays.SequenceEqual([1, 2, 3, 3, 4, 4, 5, 6, 7, 7, 7, 7]);
    expect(isEqualInContentToExpectedFlattenedArray).toBe(true);
  });

  it('can flatten one deep array into a single array', () => {
    let oneArray = [1, 2, [3, 3]];
    let flattenedArrays = oneArray.Flatten(null, 1);
    let isEqualInContentToExpectedFlattenedArray = flattenedArrays.SequenceEqual([1, 2, 3, 3]);
    expect(isEqualInContentToExpectedFlattenedArray).toBe(true);
  }); 
 

Saturday 12 June 2021

Concepts of a simple draw ink control in Windows Forms

This article will present a simple draw ink control in Windows Forms. The code is run in Linqpad and the concepts here should be easily portable to a little application. Note - there is already built in controls for Windows Forms for this (and WPF and UWP too). That is not the point of this article. The point is to display how you can use System.Reactive and Observable.FromEventPattern method to create an event source stream from
CLR events so you can build reactive applications where the source pushes updates to its target / receiver instead of traditional pull based scenario of event subscriptions. First off, we install Linqpad from: https://www.linqpad.net I used Linqpad 5 for this code, you can of course download Linqpad 6 with .Net core support, but this article is tailored for Linpad 5 and .NET Framework. After installing Linqpad 5, start it and hit F4. Choose Add Nuget. Now choose Search online and type the following four nuget packages to get started with Reactive extensions for .NET.
  • System.Reactive
  • System.Reactive.Core
  • System.Reactive.Interfaces
  • System.Reactive.Linq
Also choose Add.. and choose System.Windows.Forms. Also, choose the tab Additional Namespace Imports. Import these namespaces
  • System.Reactive
  • System.Reactive.Linq
  • System.Windows.Forms
Over to the code, first we create a Form with a PictureBox to draw onto like this in C# program:


void Main()
{
	var form = new Form();
	form.Width = 800;
	form.Height = 800;
	form.BackColor = Color.White;
	
	var canvas = new PictureBox();
	canvas.Height = 400;
	canvas.Width = 400;
	canvas.BackColor = Color.AliceBlue;
	form.Controls.Add(canvas);
    .. //more code soon


Next up we create a list of Point to add the points to. We also use Observable.FromEventPattern to track events using the System.Reactive method to create an observable from a CLR event. We then subscribe to the three events we have set up with observables and add the logic to draw anti-aliased Bezier curves. Actually, drawing a Bezier curve usually consists of the end user defining four control points, the start and end of the bezier line and two control points (for the simplest Bezier curve). However, I chose anti-aliased Bezier curves that just uses the last four points from the dragged line, since smooth Bezier curves looks way better than using DrawLine for example for simple polylines. I use GDI CreateGraphics() method of the Picturebox (this is also available on most other Windows Forms controls, including Forms, but I wanted to have the drawing restricted to the PictureBox). The full code then is the entire code snippet below:
 
 void Main()
{
	var form = new Form { Width = 800, Height = 800, BackColor = Color.White };
	var canvas = new PictureBox { Height = 400, Width = 400, BackColor = Color.AliceBlue };
	form.Controls.Add(canvas);	
    var points = new List<Point>();
	bool isDrag = false;	
	var mouseDowns = Observable.FromEventPattern<MouseEventArgs>(canvas, "MouseDown");
	var mouseUps = Observable.FromEventPattern<MouseEventArgs>(canvas, "MouseUp");
	var mouseMoves = Observable.FromEventPattern<MouseEventArgs>(canvas, "MouseMove");
	mouseDowns.Subscribe(m =>
	{
		if (m.EventArgs.Button == MouseButtons.Right)
		{
			isDrag = false;
			points.Clear();
			canvas.CreateGraphics().Clear(Color.AliceBlue);
			return;
		}
	 isDrag = true;	 
	});	
	mouseUps.Subscribe(m => {
		isDrag = false;
	});	
	mouseMoves.Subscribe(move =>  {
	 points.Add(new Point(move.EventArgs.Location.X, move.EventArgs.Location.Y));
	 if (isDrag && points.Count > 4) {
			//form.CreateGraphics().DrawLine(new Pen(Color.Blue, 10), points[points.Count - 2].X, points[points.Count - 2].Y, points[points.Count - 1].X, points[points.Count - 1].Y);
			var pt1 = new PointF(points[points.Count - 4].X, points[points.Count - 4].Y);
			var pt2 = new PointF(points[points.Count - 3].X, points[points.Count - 3].Y);
			var pt3 = new PointF(points[points.Count - 2].X, points[points.Count - 2].Y);
			var pt4 = new PointF(points[points.Count - 1].X, points[points.Count - 1].Y);			
			var graphics = canvas.CreateGraphics();
			graphics.SmoothingMode = System.Drawing.Drawing2D.SmoothingMode.AntiAlias;
			graphics.DrawBezier(new Pen(Color.Blue, 4.0f), pt1, pt2, pt3, pt4);			
		}		
	});	
	form.Show();
}


 
Linqpad/System.Reactive/GDI Windows Forms in action ! Screenshot:

I have added comments here for defining a polyline also instead of Bezier, since this also works and is quicker than the nicer Bezier curve. Maybe you want to display this on a simple device with less processing power etc. To clear the line, just hit right click button. To start drawing, just left click and drag and let go again. Now look how easy this code really is to create a simple Ink control in Windows Forms ! Of course Windows Forms today are more and more "dated" compared to younger frameworks, but it still does its job. WPF got its own built-in InkControl. But in case you want an Ink control in Windows Forms, this is an easy way of creating one and also a good Hello World to Reactive extensions. In .NET Core, the code should be really similar to the code above. Windows Forms is available with .NET Core 3.0 or newer. https://devblogs.microsoft.com/dotnet/windows-forms-designer-for-net-core-released/

Monday 7 June 2021

Json serialization using Utf8JsonReaderSerializer in .net core

.NET 5 and .net core contains a lot of new methods for Json functionality in the System.Text.Json namespace. I created a helper class for reading a file using Utf8JsonReaderSerializer and this just outputs the json to a formatted json string. With optimizations, the serialization could be done even faster. For now, I need to use a conversion between StringBuilder toString to remove last commas of arrays and properties of objects as the Utf8JsonReaderSerializer is sequential, forward-only as mentioned in the API page at: https://docs.microsoft.com/en-us/dotnet/api/system.text.json.utf8jsonreader?view=net-5.0
This is the helper method I came up with to read a file and take the way via Utf8JsonReaderSerializer:
 

using System;
using System.IO;
using System.Linq;
using System.Text;
using System.Text.Json;

namespace SystemTextJsonTestRun
{
    public static class Utf8JsonReaderSerializer
    {

        public static string ReadFile(string filePath)
        {         
            if (!File.Exists(filePath))
            {
                throw new FileNotFoundException(filePath);
            }

            var jsonBytes = File.ReadAllBytes(filePath);
            var jsonSpan = jsonBytes.AsSpan();
            var json = new Utf8JsonReader(jsonSpan);
            var sb = new StringBuilder();

            while (json.Read())
            {
                if (json.TokenType == JsonTokenType.StartObject)
                {
                    sb.Append(Environment.NewLine);
                }
                else if (json.TokenType == JsonTokenType.EndObject)
                {
                    //remove last comma added 

                    sb.RemoveLast(",");

                    sb.Append(Environment.NewLine);
                }

                if (json.CurrentDepth > 0)
                {
                    for (int i = 0; i < json.CurrentDepth; i++)
                    {
                        sb.Append(" "); //space indentation
                    }
                }

                sb.Append(GetTokenRepresentation(json));


                if (json.TokenType == JsonTokenType.EndObject || json.TokenType == JsonTokenType.EndArray)
                {
                    sb.AppendLine();
                }

                if (new[] { JsonTokenType.String, JsonTokenType.Number, JsonTokenType.Null, JsonTokenType.False,
                JsonTokenType.Number, JsonTokenType.None, JsonTokenType.True }.Contains(json.TokenType))
                {
                    sb.AppendLine(",");
                }

            }

            //remove last comma for EndObject 

            sb.RemoveLast(",");

            return sb.ToString(); 


        }


        private static string GetTokenRepresentation(Utf8JsonReader json) =>
          json.TokenType switch
          {
              JsonTokenType.StartObject => $"{{{Environment.NewLine}",
              JsonTokenType.EndObject => "},",
              JsonTokenType.StartArray => $"[{Environment.NewLine}",
              JsonTokenType.EndArray => $"]",
              JsonTokenType.PropertyName => $"\"{json.GetString()}\":",
              JsonTokenType.Comment => json.GetString(),
              JsonTokenType.String => $"\"{json.GetString()}\"",
              JsonTokenType.Number => GetNumberToString(json),
              JsonTokenType.True => json.GetBoolean().ToString().ToLower(),
              JsonTokenType.False => json.GetBoolean().ToString().ToLower(),
              JsonTokenType.Null => string.Empty,
              _ => "Unknown Json token type"
          };

        //TODO: Use the Try methods of the Utf8JsonReader more than trying and failing here 

        private static string GetNumberToString(Utf8JsonReader json)
        {
            try
            {
                if (int.TryParse(json.GetInt32().ToString(), out var res))
                    return res.ToString();
            }
            catch
            {
                try
                {
                    if (float.TryParse(json.GetSingle().ToString(), out var resFloat))
                        return resFloat.ToString();
                }
                catch
                {
                    try
                    {
                        if (decimal.TryParse(json.GetDouble().ToString(), out var resDes))
                            return resDes.ToString();
                    }
                    catch
                    {
                        return "?";
                    }
                }
            }
            return $"?"; //fallback to a string if not possible to deduce the type
        }

    }
}

  

The json file I tested the code with inputted came out again as this string:

{
 "courseName": "Build Your Own Application Framework",
 "language": "C#",
 "author":
 {
  "firstName":  "Matt",
  "lastName":  "Honeycutt"

 },
 "publishedAt": "2012-03-13T12:30:00.000Z",
 "publishedYear": 2014,
 "isActive": true,
 "isRetired": false,
 "tags": [
  "aspnet",
  "C#",
  "dotnet"
 ]

}

This code validates against Json Lint also: https://jsonlint.com Now why even bother parsing a Json file just to output the file again to a json string? Well, first of all, we use a very fast parser Utf8JsonReader from .NET and we can for example do various processing along the forward-only sequential processing and formatting indented the file. Utf8JsonReader will also validate the json document strictly to the Json specification - RFC 8259. Hence, we can get validation for free here to by catching any errors and returning true or false in method that scans this file by adding a method for this looking at the json.Read() method (if it returns false) or catching JsonException if a node of the json document does not validate. Also, a low level analysis of the Utf8JsonReader let's you see which different tokens of the json document structure .NET provides. We could transform the document or add specific formatting and so on by altering the code displayed here. To run the code test with a sample json document like this:

   class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Utf8JsonReader sample");

            string json = Utf8JsonReaderSerializer.ReadFile("sample.json");
            string tempFile = Path.ChangeExtension(Path.GetTempFileName(), "json"); 
            File.WriteAllText(tempFile, json);
            Console.WriteLine($"Json file read and processed result in location: {tempFile}");
            Console.WriteLine($"Json file contents: {Environment.NewLine}{json}");

        
        }

I have added the code for this here: https://github.com/toreaurstadboss/Utf8DataJsonReaderTest/

Sunday 25 April 2021

Making NUnit tests run in Team City for NUnit 3.x

Team City has several bugs when it comes to running NUnit tests. The following guide shows how you can prepare the Team City build agent to run NUnit 3.x tests. We need first to install NUnit Console runner Tips around this was found in the following Stack Overflow thread: This is also mentioned in the documentation of Team City: First off, add two Command line steps and add the two commands into each step - this step can be run at the start of the pipeline in Team City.


%teamcity.tool.NuGet.CommandLine.DEFAULT%\tools\nuget.exe install NUnit.Console -version 3.10.0 -o packages  -ExcludeVersion -OutputDirectory %system.teamcity.build.tempDir%\NUnit %teamcity.tool.NuGet.CommandLine.DEFAULT%\tools\nuget.exe install NUnit.Extension.NUnitProjectLoader -version 3.6.0 -o packages
The following Nuget packages for NUnit was used:
  • NUnit 3.2.0
  • NUnit.ConsoleRunner 3.10.0
  • NUnit.Extension.NUnitProjectLoader 3.6.0
  • NUnit.Extension.TeamCityEventListener 1.0.7
  • NUnit3TestAdapter 3.16.1
Inside the NUnit runner type step, configure also the NUnit console path: Use this path:
packages\NUnit.ConsoleRunner.3.8.0\tools\nunit3-console.exe For the testassemblies make sure you use a path like this: **\bin\%BuildConfiguration%\*.Test.dll Add the %BuildConfiguration% parameter and set it to: Debug
More tips here: https://stackoverflow.com/questions/57953724/nunit-teamcity-process-exited-with-code-4

And here:
https://stackoverflow.com/questions/36996564/nunit-3-2-1-teamcity-could-not-load-file-or-assembly-nunit-framework

Sunday 18 April 2021

Implementing a Strip method with regex in C#

This article will present a Strip method that accepts a Regex which defines the pattern of allowed characters. It is similar to Regex Replace, but it works in the inverted way. Instead of removing the chars matching the pattern in Regex.Replace, this utility method instead lets you define the allowed chars, i.e. these chars defined in this regex are the chars I want to keep. First off we define the utility method, as an extension method.
 

        /// <summary>
        /// Strips away every character not defined in the provided regex <paramref name="allowedChars"/>
        /// </summary>
        /// <param name="s">Input string</param>
        /// <param name="allowedChars">The allowed characters defined in a Regex with pattern, for example: [A-z|0-9]+/</param>
        /// <returns>Input string with only the allowed characters</returns>
        public static string Strip(this string s, Regex allowedChars)
        {
            if (s == null)
            {
                return s;
            }
            if (allowedChars == null)
            {
                return string.Empty;
            }
            Match match = Regex.Match(s, allowedChars.ToString());
            List<char> allowedAlphabet = new List<char>();
            while (match.Success)
            {
                if (match.Success)
                {
                    for (int i = 0; i < match.Groups.Count; i++)
                    {
                        allowedAlphabet.AddRange(match.Groups[i].Value.ToCharArray());
                    }
                }
                match = match.NextMatch();
            }
            return new string(s.Where(ch => allowedAlphabet.Contains(ch)).ToArray());
        }
        
          
Here are some tests that tests out this Strip method:
 
 
 	 	[Test]
        [TestCase("abc123abc", "[A-z]+", "abcabc")]
        [TestCase("abc123def456", "[0-9]+", "123456")]
     	[TestCase("The F-32 Lightning II is a newer generation fighter jets than the F-16 Fighting Falcon", "[0-9]+", "3216")]
		[TestCase("Here are some Norwegian letters : ÆØÅ and in lowercase: æøå", "[æ|ø|å]", "æøå")]
		public void TestStripWithRegex(string input, string regexString, string expectedOutput)
        {
            var regex = new Regex(regexString);
            input.Strip(regex).Should().Be(expectedOutput);
        }
 
 

Monday 1 March 2021

Implementing ToDictionary in Typescript

In this article I will present some code I just did in my SimpleTsLinq library, which you can easily install using Npm. The library is here on Npmjs.com : The ToDictionary method looks like this:

  
  if (!Array.prototype.ToDictionary) {
  Array.prototype.ToDictionary = function <T>(keySelector: (arg: T) => any): any {
    let hash = {};
    this.map(item => {
      let key = keySelector(item);
      if (!(key in hash)) {
        hash[key] = item;
      }
      else {
        if (!(Array.isArray(hash[key]))) {
          hash[key] = [hash[key]];
        }
        hash[key].push(item);
      }
    });
    return hash;
  }
}
  

Here is a unit test (spec) for this method :

  
    it('can apply method ToDictionary on an array, allowing specificaton of a key selector for the dictionary object', () => {
    let heroes = [{ name: "Han Solo", age: 47, gender: "M" }, { name: "Leia", age: 29, gender: "F" }, { name: "Luke", age: 24, gender: "M" }, { name: "Lando", age: 47, gender: "M" }];
    let dictionaryOfHeroes = heroes.ToDictionary<Hero>(x => x.gender);

    let expectedDictionary = {
      "F": {
        name: "Leia", age: 29, gender: "F"
      },
      "M": [
        { name: "Han Solo", age: 47, gender: "M" },
        { name: "Luke", age: 24, gender: "M" },
        { name: "Lando", age: 47, gender: "M" }
      ]
    };
    expect(dictionaryOfHeroes).toEqual(expectedDictionary);
  });
  
  

You can also test out this library using Npm RunKit here: We can make a dictionary with different keys, image example:

Friday 5 February 2021

Overcoming limitations in Contains in Entity Framework

Entity Framework will hit a performance penalty bottleneck or crash if Contains contains a too large of a list. Here is how you can avoid this, using Marc Gravell's excellent approach in this. I am including some tests of this. I also suggest you consider LinqKit to use expandable queries to make this all work. First off, this class contains the extension methods for Entity Framework for this:
 
 
 public class EntityExtensions {
 
 /// <summary>
        /// This method overcomes a weakness with Entity Framework with Contains where you can partition the values to look for into 
        /// blocks or partitions, it is modeled after Marc Gravell's answer here:
        /// https://stackoverflow.com/a/568771/741368
        /// Entity Framework hits a limit of 2100 parameter limit in the DB but probably comes into trouble before this limit as even
        /// queries with several 100 parameters are slow.
        /// </summary>
        /// <typeparam name="T"></typeparam>
        /// <typeparam name="TValue"></typeparam>
        /// <param name="source">Source, for example DbSet (table)</param>
        /// <param name="selector">Selector, key selector</param>
        /// <param name="blockSize">Size of blocks (chunks/partitions)</param>
        /// <param name="values">Values as parameters</param>
        /// 
        /// <example>
        ///   /// <[!CDATA[
        /// /// The following EF query will hit a performance penalty or time out if EF gets a too large list of operationids:
        /// ///
        /// /// var patients = context.Patients.Where(p => operationsIds.Contains(p.OperationId)).Select(p => new {
        /// ///  p.OperationId,
        /// ///  p.
        /// /// });
        /// ///
        /// /// 
        /// /// var patients = context.Patients.AsExpandable().InRange(p => p.OperationId, 1000, operationIds)
        /// //.Select(p => new
        /// //{
        /// //    p.OperationId,
        /// //    p.IsDaytimeSurgery
        /// //}).ToList();
        /// //]]
        /// </example>
        /// <returns></returns>
        public static IEnumerable<T> InRange<T, TValue>(
                this IQueryable<T> source,
                Expression<Func<T, TValue>> selector,
                int blockSize,
                IEnumerable<TValue> values)
        {
            MethodInfo method = null;
          
            foreach (MethodInfo tmp in typeof(Enumerable).GetMethods(
                    BindingFlags.Public | BindingFlags.Static))
            {
                if (tmp.Name == "Contains" && tmp.IsGenericMethodDefinition
                        && tmp.GetParameters().Length == 2)
                {
                    method = tmp.MakeGenericMethod(typeof(TValue));
                    break;
                }
            }

            if (method == null) throw new InvalidOperationException(
                   "Unable to locate Contains");
            foreach (TValue[] block in values.GetBlocks(blockSize))
            {
                var row = Expression.Parameter(typeof(T), "row");
                var member = Expression.Invoke(selector, row);
                var keys = Expression.Constant(block, typeof(TValue[]));
                var predicate = Expression.Call(method, keys, member);
                var lambda = Expression.Lambda<Func<T, bool>>(
                      predicate, row);
                foreach (T record in source.Where(lambda))
                {
                    yield return record;
                }
            }

        }

        /// <summary>
        /// Similar to Chunk, it partitions the IEnumerable source and returns the chunks or blocks by given blocksize. The last block can have variable length
        /// between 0 to blocksize since the IEnumerable can have of course variable size not evenly divided by blocksize. 
        /// </summary>
        /// <typeparam name="T"></typeparam>
        /// <param name="source"></param>
        /// <param name="blockSize"></param>
        /// <returns></returns>
        public static IEnumerable<T[]> GetBlocks<T>(
                this IEnumerable<T> source, int blockSize)
        {
            List<T> list = new List<T>(blockSize);
            foreach (T item in source)
            {
                list.Add(item);
                if (list.Count == blockSize)
                {
                    yield return list.ToArray();
                    list.Clear();
                }
            }
            if (list.Count > 0)
            {
                yield return list.ToArray();
            }
        }
        
  }

 
 
 
Linqkit allows us to rewrite queries for EF using expression trees. One class is ExpandableQuery. See the links here for further info about Linqkit and Linq-Expand.
 
  	/// <summary>Refer to http://www.albahari.com/nutshell/linqkit.html and
	/// http://tomasp.net/blog/linq-expand.aspx for more information.</summary>
	public static class Extensions
	{
		public static IQueryable<T> AsExpandable<T> (this IQueryable<T> query)
		{
			if (query is ExpandableQuery<T>) return (ExpandableQuery<T>)query;
			return new ExpandableQuery<T> (query);
		}
 
This all seems to look a bit cryptic, so lets see an integration test of mine instead:
 


        [Test]
        [Category(TestCategories.IntegrationTest)]
        public void GetDataChunkedDoesNotFail()
        {
            using (var context = DbContextManager.ScopedOpPlanDataContext)
            {
                int[] operationalUnitIds = new int[]{ 107455, 105431, 107646, 107846 };
                var reportItems = context.OperationalUnits.AsExpandable().InRange(ou => ou.FreshOrganizationalUnitId, 2, operationalUnitIds).ToList();
                Assert.IsNotNull(reportItems);           
                CollectionAssert.IsNotEmpty(reportItems); 
            }
        }

 
 
 
This shows how to use the InRange method of Marc Gravell. We use the AsExpandable method to allow us to hack into the expression tree of Entity Framework and the InRange method allows us to partition the work for EF. We do not know the siz of operational unit ids (usually it is low and another entity - operation Ids is of variable length and will in production blow up since we in some cases surpass the 2100 limit of Contains). And as I said before, Entity Framework will hit a performance bottleneck before 2100 parameteters are sent into the Contains method. This way of fixing it up will allow you to get stable running code in production again against large data and variable length. This code is tested with Entity Framework 6.2.0. Another article considers performance considerations for Contains and different approaches here: https://www.toptal.com/dot-net/entity-framework-performance-using-contains IMHO this approach has proven stable in a production environment for several years with large data and can be considered a stable workaround for EF slow Contains performance. I have made the LinqKit fork LinqKit.AsyncSupport available on Nuget here now: https://www.nuget.org/packages/ToreAurstadIt.LinqKit.AsyncSupport/1.1.0 This makes it possible to perform Async calls and expandable queries, i.e. queries with inline method calls for example. The nuget package now also sports symbol package for easier debugging experience. The source code for LinqKit.AsyncSupport is available here: https://github.com/toreaurstadboss/LinqKit.AsyncSupport