Overcoming limitations in Contains in Entity Framework

Entity Framework will hit a performance penalty bottleneck or crash if Contains contains a too large of a list. Here is how you can avoid this, using Marc Gravell's excellent approach in this. I am including some tests of this. I also suggest you consider LinqKit to use expandable queries to make this all work. First off, this class contains the extension methods for Entity Framework for this:

 
 
 public class EntityExtensions {
 
 /// <summary>
        /// This method overcomes a weakness with Entity Framework with Contains where you can partition the values to look for into 
        /// blocks or partitions, it is modeled after Marc Gravell's answer here:
        /// https://stackoverflow.com/a/568771/741368
        /// Entity Framework hits a limit of 2100 parameter limit in the DB but probably comes into trouble before this limit as even
        /// queries with several 100 parameters are slow.
        /// </summary>
        /// <typeparam name="T"></typeparam>
        /// <typeparam name="TValue"></typeparam>
        /// <param name="source">Source, for example DbSet (table)</param>
        /// <param name="selector">Selector, key selector</param>
        /// <param name="blockSize">Size of blocks (chunks/partitions)</param>
        /// <param name="values">Values as parameters</param>
        /// 
        /// <example>
        ///   /// <[!CDATA[
        /// /// The following EF query will hit a performance penalty or time out if EF gets a too large list of operationids:
        /// ///
        /// /// var patients = context.Patients.Where(p => operationsIds.Contains(p.OperationId)).Select(p => new {
        /// ///  p.OperationId,
        /// ///  p.
        /// /// });
        /// ///
        /// /// 
        /// /// var patients = context.Patients.AsExpandable().InRange(p => p.OperationId, 1000, operationIds)
        /// //.Select(p => new
        /// //{
        /// //    p.OperationId,
        /// //    p.IsDaytimeSurgery
        /// //}).ToList();
        /// //]]
        /// </example>
        /// <returns></returns>
        public static IEnumerable<T> InRange<T, TValue>(
                this IQueryable<T> source,
                Expression<Func<T, TValue>> selector,
                int blockSize,
                IEnumerable<TValue> values)
        {
            MethodInfo method = null;
          
            foreach (MethodInfo tmp in typeof(Enumerable).GetMethods(
                    BindingFlags.Public | BindingFlags.Static))
            {
                if (tmp.Name == "Contains" && tmp.IsGenericMethodDefinition
                        && tmp.GetParameters().Length == 2)
                {
                    method = tmp.MakeGenericMethod(typeof(TValue));
                    break;
                }
            }

            if (method == null) throw new InvalidOperationException(
                   "Unable to locate Contains");
            foreach (TValue[] block in values.GetBlocks(blockSize))
            {
                var row = Expression.Parameter(typeof(T), "row");
                var member = Expression.Invoke(selector, row);
                var keys = Expression.Constant(block, typeof(TValue[]));
                var predicate = Expression.Call(method, keys, member);
                var lambda = Expression.Lambda<Func<T, bool>>(
                      predicate, row);
                foreach (T record in source.Where(lambda))
                {
                    yield return record;
                }
            }

        }

        /// <summary>
        /// Similar to Chunk, it partitions the IEnumerable source and returns the chunks or blocks by given blocksize. The last block can have variable length
        /// between 0 to blocksize since the IEnumerable can have of course variable size not evenly divided by blocksize. 
        /// </summary>
        /// <typeparam name="T"></typeparam>
        /// <param name="source"></param>
        /// <param name="blockSize"></param>
        /// <returns></returns>
        public static IEnumerable<T[]> GetBlocks<T>(
                this IEnumerable<T> source, int blockSize)
        {
            List<T> list = new List<T>(blockSize);
            foreach (T item in source)
            {
                list.Add(item);
                if (list.Count == blockSize)
                {
                    yield return list.ToArray();
                    list.Clear();
                }
            }
            if (list.Count > 0)
            {
                yield return list.ToArray();
            }
        }
        
  }

 
 
 

Linqkit allows us to rewrite queries for EF using expression trees. One class is ExpandableQuery. See the links here for further info about Linqkit and Linq-Expand.

 
  	/// <summary>Refer to http://www.albahari.com/nutshell/linqkit.html and
	/// http://tomasp.net/blog/linq-expand.aspx for more information.</summary>
	public static class Extensions
	{
		public static IQueryable<T> AsExpandable<T> (this IQueryable<T> query)
		{
			if (query is ExpandableQuery<T>) return (ExpandableQuery<T>)query;
			return new ExpandableQuery<T> (query);
		}
 

This all seems to look a bit cryptic, so lets see an integration test of mine instead:

 


        [Test]
        [Category(TestCategories.IntegrationTest)]
        public void GetDataChunkedDoesNotFail()
        {
            using (var context = DbContextManager.ScopedOpPlanDataContext)
            {
                int[] operationalUnitIds = new int[]{ 107455, 105431, 107646, 107846 };
                var reportItems = context.OperationalUnits.AsExpandable().InRange(ou => ou.FreshOrganizationalUnitId, 2, operationalUnitIds).ToList();
                Assert.IsNotNull(reportItems);           
                CollectionAssert.IsNotEmpty(reportItems); 
            }
        }

 
 
 

This shows how to use the InRange method of Marc Gravell. We use the AsExpandable method to allow us to hack into the expression tree of Entity Framework and the InRange method allows us to partition the work for EF. We do not know the siz of operational unit ids (usually it is low and another entity - operation Ids is of variable length and will in production blow up since we in some cases surpass the 2100 limit of Contains). And as I said before, Entity Framework will hit a performance bottleneck before 2100 parameteters are sent into the Contains method. This way of fixing it up will allow you to get stable running code in production again against large data and variable length. This code is tested with Entity Framework 6.2.0. Another article considers performance considerations for Contains and different approaches here: https://www.toptal.com/dot-net/entity-framework-performance-using-contains IMHO this approach has proven stable in a production environment for several years with large data and can be considered a stable workaround for EF slow Contains performance. I have made the LinqKit fork LinqKit.AsyncSupport available on Nuget here now: https://www.nuget.org/packages/ToreAurstadIt.LinqKit.AsyncSupport/1.1.0 This makes it possible to perform Async calls and expandable queries, i.e. queries with inline method calls for example. The nuget package now also sports symbol package for easier debugging experience. The source code for LinqKit.AsyncSupport is available here: https://github.com/toreaurstadboss/LinqKit.AsyncSupport

Simple property grid in Blazor

This artile will present a simple property grid in Blazor I have made. The component relies on standard stuff like Bootstrap, jQuery, Twitter Bootstrap and Font Awesome. But the repo url shown here links to the Github repo of mine which can be easily forked if you want to add features (such as editing capabilities). The component already supports nested levels, so if the object you inspect has a hierarchical structure, this is shown in this Blazor component. Having a component to inspect objects in Blazor is great as Blazor lacks inspect tools (since the app is compiled into a web assembly, we cannot easily inspect state of objects in the app other than the DOM and Javascript objects. With this component we can get basic inspection support to inspect state of the object in the app you desire to inspect). The Github repo contains also a bundled application which uses the component and shows a sample use-case (also shown in Gif video below). I have tested the component with three levels of depth for a sample object (included in the repo). The component is available here on my Github repo:

 
 git clone https://github.com/toreaurstadboss/BlazorPropertyGrid/tree/main/BlazorPropertyGrid
 

https://github.com/toreaurstadboss/BlazorPropertyGrid/tree/main/BlazorPropertyGrid

The component consists of two components where one of them is used in a recursive manner to support nested object structure. The top level component got this code-behind class.

PropertyGridComponentBase.cs


using System.Collections.Generic;
using System.Reflection;
using Microsoft.AspNetCore.Components;
using Microsoft.AspNetCore.Components.Web;
using Microsoft.JSInterop;

namespace BlazorPropertyGridComponents.Components
{
    public class PropertyGridComponentBase : ComponentBase
    {

        [Inject]
        public IJSRuntime JsRuntime { get; set; }

        [Parameter] public object DataContext { get; set; }

        public Dictionary<string, PropertyInfoAtLevelNodeComponent> Props { get; set; }


        public PropertyGridComponentBase()
        {
            Props = new Dictionary<string, PropertyInfoAtLevelNodeComponent>();
        }


        protected override void OnParametersSet()
        {
            Props.Clear();
            if (DataContext == null)
                return;
          
            Props["ROOT"] = MapPropertiesOfDataContext(string.Empty, DataContext, null);

            StateHasChanged();
        }

        private bool IsNestedProperty(PropertyInfo pi) =>
            pi.PropertyType.IsClass && pi.PropertyType.Namespace != "System";

        private PropertyInfoAtLevelNodeComponent MapPropertiesOfDataContext(string propertyPath, object parentObject,
            PropertyInfo currentProp)
        {
            if (parentObject == null)
                return null;

            var publicProperties = parentObject.GetType()
                .GetProperties(BindingFlags.Public | BindingFlags.Instance);

            var propertyNode = new PropertyInfoAtLevelNodeComponent
            {
                PropertyName = currentProp?.Name ?? "ROOT",
                PropertyValue = parentObject,
                PropertyType = parentObject.GetType(),
                FullPropertyPath = TrimFullPropertyPath($"{propertyPath}.{currentProp?.Name}") ?? "ROOT",
                IsClass = parentObject.GetType().IsClass && parentObject.GetType().Namespace != "System"
            };

            foreach (var p in publicProperties)
            {
                var propertyValue = p.GetValue(parentObject, null); 

                if (!IsNestedProperty(p))
                {
                    propertyNode.SubProperties.Add(p.Name, new PropertyInfoAtLevelNodeComponent
                        {
                            IsClass = false,
                            FullPropertyPath = TrimFullPropertyPath($"{propertyPath}.{p.Name}"),
                            PropertyName = p.Name,
                            PropertyValue = propertyValue,
                            PropertyType = p.PropertyType
                            //note - SubProperties are default empty if not nested property of course.
                        }
                    );
                }
                else
                {
                    //we need to add the sub property but recurse also call to fetch the nested properties 
                    propertyNode.SubProperties.Add(p.Name, new PropertyInfoAtLevelNodeComponent
                        {
                            IsClass = true,
                            FullPropertyPath = propertyPath + p.Name,
                            PropertyName = p.Name,
                            PropertyValue = MapPropertiesOfDataContext(TrimFullPropertyPath($"{propertyPath}.{p.Name}"), propertyValue, p),
                            PropertyType = p.PropertyType
                            //note - SubProperties are default empty if not nested property of course.
                        }
                    );
                }
            }

            return propertyNode;
        }

        protected void toggleExpandButton(MouseEventArgs e, string buttonId)
        {
            JsRuntime.InvokeVoidAsync("toggleExpandButton", buttonId);
        }

        private string TrimFullPropertyPath(string fullpropertypath)
        {
            if (string.IsNullOrEmpty(fullpropertypath))
                return fullpropertypath;
            return fullpropertypath.TrimStart('.').TrimEnd('.');
        }

    }

}

And its razor file looks like this:

PropertyGridComponentBase.razor

@inherits PropertyGridComponentBase
@using BlazorPropertyGridComponents.Components

<table class="table table-striped col-md-4 col-lg-3 col-sm-6">
    <thead>
        <tr>
            <th scope="col">Property</th>
            <th scope="col">Value</th>
        </tr>
    </thead>
    <tbody>
        @foreach (KeyValuePair<string, PropertyInfoAtLevelNodeComponent> prop in Props)
        {
            @if (!prop.Value.IsClass)
            {
               @* <tr>
                    <td>@prop.Key</td>
                    <td>@prop.Value</td>
                </tr>*@
            }
            else
            {
                
                var currentNestedDiv = "currentDiv_" + prop.Key;
                var currentProp = prop.Value.PropertyValue;
                //must be a nested class property
                <tr>
                    <td colspan="2">
              
                        <button type="button" id="@prop.Key" class="btn btn-info fas fa-minus"  @onclick="(e) => toggleExpandButton(e,prop.Key)"  data-toggle="collapse" data-target="#@currentNestedDiv">
                        </button>
                        <div id="@currentNestedDiv" class="collapse show">
                           <PropertyRowComponent Depth="1" PropertyInfoAtLevel="@prop.Value" />
                        </div>
                        </td>
                        </tr>
            }
        }
    </tbody>
</table>




@code {
}

We also have this helper class to model each property in the nested structure:

PropertyInfoAtLevelNodeComponent.cs

using System;
using System.Collections.Generic;

namespace BlazorPropertyGridComponents.Components
{

    /// <summary>
    /// Node class for hierarchical structure of property info for an object of given object graph structure.
    /// </summary>
    public class PropertyInfoAtLevelNodeComponent
    {
        public PropertyInfoAtLevelNodeComponent()
        {
            SubProperties = new Dictionary<string, PropertyInfoAtLevelNodeComponent>();
        }

        public string PropertyName { get; set; }
        public object PropertyValue { get; set; }
        public Type PropertyType { get; set; }
        public Dictionary<string, PropertyInfoAtLevelNodeComponent> SubProperties { get; private set; }
        public string FullPropertyPath { get; set; }
        public bool IsClass { get; set; }
    }

}

Our lower component used by the top component code-behind looks like this:

PropertyRowComponentBase.cs

using System.Collections.Generic;
using Microsoft.AspNetCore.Components;
using Microsoft.AspNetCore.Components.Web;
using Microsoft.JSInterop;

namespace BlazorPropertyGridComponents.Components
{
    public class PropertyRowComponentBase : ComponentBase
    {
        public PropertyRowComponentBase()
        {
            DisplayedFullPropertyPaths = new List<string>();
        }
        [Parameter]
        public PropertyInfoAtLevelNodeComponent PropertyInfoAtLevel { get; set; }

        [Parameter]
        public int Depth { get; set; }

        [Parameter]
        public List<string> DisplayedFullPropertyPaths { get; set; }

        [Inject]
        protected IJSRuntime JsRunTime { get; set; }

        protected void toggleExpandButton(MouseEventArgs e, string buttonId)
        {
            JsRunTime.InvokeVoidAsync("toggleExpandButton", buttonId);
        }

    }
}

The razor file looks like this:

PropertyRowComponent.razor


@using  BlazorPropertyGridComponents.Components
@inherits PropertyRowComponentBase

@foreach (var item in PropertyInfoAtLevel.SubProperties.Keys)
{
    var propertyInfoAtLevel = PropertyInfoAtLevel.SubProperties[item];
    if (propertyInfoAtLevel != null)
    {
     @*   if (DisplayedFullPropertyPaths.Contains(propertyInfoAtLevel.FullPropertyPath)){
          continue; //the property is already displayed.
      }*@
      DisplayedFullPropertyPaths.Add(propertyInfoAtLevel.FullPropertyPath); 
      
 @*        <span class="text-white bg-dark">@propertyInfoAtLevel.FullPropertyPath</span>*@

@*  <em>
@propertyInfoAtLevel
</em>*@
    }

    if (!propertyInfoAtLevel.PropertyType.IsClass || propertyInfoAtLevel.PropertyType.Namespace.StartsWith("System"))
    {
<tr>
    <td>
        <span title="@propertyInfoAtLevel.FullPropertyPath" class="font-weight-bold">@propertyInfoAtLevel.PropertyName</span>
    </td>
    <td>
        <span>@propertyInfoAtLevel.PropertyValue</span>
       
    </td>
</tr>
}
    else if (propertyInfoAtLevel.PropertyValue != null && propertyInfoAtLevel.PropertyValue is PropertyInfoAtLevelNodeComponent)
    {
        var nestedLevel = (PropertyInfoAtLevelNodeComponent)propertyInfoAtLevel.PropertyValue;
        var collapseOrNotCssClass = Depth == 0 ? "collapse show" : "collapse";
        var curDepth = Depth + 1;

        collapseOrNotCssClass += " depth" + Depth;


 var currentNestedDiv = "collapsingdiv_" + propertyInfoAtLevel.PropertyName;

 //must be a nested class property
<tr>
    <td colspan="2">
        <span>@propertyInfoAtLevel.PropertyName</span>
        <button id="@propertyInfoAtLevel.FullPropertyPath" type="button" @onclick="(e) => toggleExpandButton(e,propertyInfoAtLevel.FullPropertyPath)" class="fas btn btn-info fa-plus" data-toggle="collapse" data-target="#@currentNestedDiv"></button>
        <div id="@currentNestedDiv" class="@collapseOrNotCssClass">
            <PropertyRowComponent PropertyInfoAtLevel="@nestedLevel" Depth="@curDepth" />
        </div>
    </td>
</tr>
}
}


@code {
}

More fun with Blazor - creating a folder viewer in 10 minutes

 
   git clone https://github.com/toreaurstadboss/BlazorLiveReloadSample.git 
 

Blazor is very easy to use! I spent 10 minutes to create a simple folder viewer here now.

First off, the Blazor razor component looks like this:

FilewView.razor
 
 
 @inject IJSRuntime jsRuntime;
<h3>FileView</h3>

@foreach (var folder in folders)
{
    var depthOfFolder = folder.Split('\\').Count();
    <p style="cursor:pointer" @onclick="() => openFolder(folder)">
    @for (int i = 0; i < depthOfFolder; i++)
    {
        <span style="margin-left:10px"></span>
    }
    <i style="color:orange;cursor:pointer" class="fa fa-folder" /> @folder </p>
}


@code {

    List<string> folders = Directory.GetDirectories("/").ToList();

    private void openFolder(string folder)
    {
        jsRuntime.InvokeVoidAsync("log", folder);
        string[] subfolders = Directory.GetDirectories(folder);
        Console.WriteLine(folder);
        int folderIndex = folders.IndexOf(folder);
        folders.InsertRange(folderIndex, subfolders);
    }

}

 
 
 

We add to the solution also Font-awesome via right click solution explorer and choose Add => Client-Side Library. Search for 'font-awesome' Choose Font Awesome and add all files to be added to the lib/font-awesome folder of wwwroot. Then at the bottom of _Host.cshtml we add:

 
_Host.cshtml
   
 <link rel="stylesheet" href="~/lib/font-awesome/css/all.css" />
 

Now we have access to the Font Awesome icons.