Saturday, 4 June 2022

Making use of extension methods to extract data from FHIR bundles

This article shows some extension methods to extract data from FHIR bundles, where FHIR stands for Fast Healthcare Interoperability Resources. The standard is used as a global or country specific standard with its own
variants. It is also a standard that allows for extensibility and its goal is to define interoperability and an information model definining resources which are then comprised of smaller elements which can define different kinds of information.
FHIR also defines API standards and is defined in different formats such as XML and json. We will look into some example extension methods for retrieving data deep inside a FHIR Bundle. A bundle is a top level data which got a lot of components in a hierarchical structure as we define the data in XML or JSON for example, i.e. a tree structure. Let us say that we want to retrieve data like medications a pasient is taking as his or her DDD (Defined Daily Dosage). We want to be able to find a medication statement inside our bundle and then retrieve the medication dosage quantity. We know the unit is measured in micrograms (ug) and the medication (drug) is called Fentanyl. Here is a property getter with logic to retrieve this value.
 
   public int? Fentanyl
   {
            get
            {
                var dosageQuantity = _bundle.SearchMedicationStatements("http://someacme.no/fhir/MedicationStatement/")
                    ?.GetMedicationDosageQuantity("Fentanyl", "ug");
                //value is already a decimal? data type and must be parsed 
                if (int.TryParse(dosageQuantity?.Value?.ToString(), out var dosageQuantityParsed))
                {
                    return dosageQuantityParsed;
                }
                return null;
            }
    }
 

We have these two extension methods to help us with retrieving the data :
 
      
        public static List<MedicationStatement>? SearchMedicationStatements(this Bundle bundle, string resourcePath)
        {
            var medicationStatementsMatching = bundle?.Entry?.Where(e => e.FullUrl.StartsWith(resourcePath))?.Select(m => m.Resource)?.OfType<MedicationStatement>()?.ToList();
            return medicationStatementsMatching;
        }

        public static Dosage? GetMedicationDosageDosage(this List<MedicationStatement> medicationStatements, string displayText)
        {
            //find dosage with given display text 

            foreach (var medicationStatement in medicationStatements)
            {
                var medicationStatementMedication = medicationStatement?.Medication as CodeableConcept; 
                if (medicationStatementMedication == null)
                {
                    continue; 
                }
                var medicationCoding = medicationStatementMedication?.Coding?.FirstOrDefault(med => med.Display?.Equals(displayText, StringComparison.InvariantCultureIgnoreCase) == true);  
                if (medicationCoding != null)
                {
                    var quantity = medicationStatement?.Dosage?.FirstOrDefault();
                    return quantity; 
                }           
            }

            return null;
        }
 
 
And our unit test will then be very simply with some static Fhir bundle data like this :
 
 
    [TestFixture]
    public class SomeAcmeManagerTests
    {
        private Bundle? _bundle = new();
        private SomeAcmeColoscopyDomainModel? _domainModel;

        [SetUp]
        public void TestInitialize()
        {
            _bundle = new FhirJsonParser().Parse<Bundle>(File.ReadAllText(@"TestData/JSON_someacme.json"));
            _domainModel = new SomeAcmeColoscopyDomainModel(_bundle, _metadataVersionId);
        }
 
        [Test]
        public void Map_Fhir_Bundle_To_Property_Fentanyl()
        {
            _domainModel?.Fentanyl.Should().Be(2);
        }
 
Now, we have used some classes here as you have seen called FhirJsonParser and MedicationStatements. These classes and functionality is available in some selected nuget packages :
 
  <PackageReference Include="Hl7.Fhir.R4" Version="4.0.0" />
  <PackageReference Include="Hl7.Fhir.Serialization" Version="4.0.0" />
  <PackageReference Include="Hl7.Fhir.Support" Version="4.0.0" />
  <PackageReference Include="Hl7.Fhir.Support.Poco" Version="4.0.0" />
 
This packages are licensed under BSD-3 license and are 'free' as long as you include the copyright notice. See this url for more info - it is the HL7 FHIR SDK for Microsoft .net platform. As you see we have additional nuget packages for (de)serialization and Poco objects. We also have common interfaces and classes in the HL7.Fhir.Support and HL7.Fhir.Poco nuget packages. This makes it way easier to work with a large bundle of Fhir data. Now, about sample data - this is often given in a .json file by the for example another organization, that you want to integrate with. FHIR is about interoperability and a common understanding of different health information systems via a common standard. The sample data starts with these data at the top - a bundle and a diagnosticreport. Of course, FHIR is a very large standard and which data you work against will vary a lot.
 
 {
  "resourceType": "Bundle",
  "meta": {
    "profile": [
      "http://someacmeregistries.no/fhir/StructureDefinition/colonoscopyreport-bundle-someacme"
    ]
  },
  "identifier": {
    "system": "http://someacmeotherorg.no/fhir/NamingSystem/colonoscopy-report-id",
    "value": "IdPlaceholder"
  },
  "type": "collection",
  "timestamp": "2022-05-10T10:26:40.6221425+02:00",
  "entry": [
    {
      "fullUrl": "http://somethirdacme.no/fhir/DiagnosticReport/c83b2f53-2d01-46e7-aed4-703396d5433f",
      "resource": {
        "resourceType": "DiagnosticReport",
        "meta": {
          "profile": [
            "http://someacmeregistries.no/fhir/StructureDefinition/colonoscopyreport-diagnosticreport-gastronet"
          ]
        },
        "status": "final",
        "code": {
          "coding": [
            {
              "system": "http://snomed.info/sct",
              "code": "73761001",
              "display": "Koloskopi"
            }
          ]
        },
        "subject": {
          "reference": "http://someacme.no/fhir/Patient/84c51e6c-020c-469b-b6c8-a6e9b2db6ff6"
        },
 
 
As you can see we also have something called Snomed SCT codes in our data. The meaning of these codes can be looked up online. International edition is here: https://browser.ihtsdotools.org/? Our medication drug statment for Fentanyl is defined further into the FHIR json bundle.


    {
      "fullUrl": "http://someacme.no/fhir/MedicationStatement/ae48d3bf-c289-4290-b7ed-78ef7bb6f1b5",
      "resource": {
        "resourceType": "MedicationStatement",
        "meta": {
          "profile": [
            "http://someame.no/fhir/StructureDefinition/colonoscopyreport-medicationstatement-gastronet"
          ]
        },
        "partOf": [
          {
            "reference": "http://someacme.no/fhir/Procedure/14f9f2e3-0d53-46c9-936f-3f0f85dd8cce"
          }
        ],
        "status": "active",
        "medicationCodeableConcept": {
          "coding": [
            {
              "system": "http://snomed.info/sct",
              "code": "373492002",
              "display": "Fentanyl"
            }
          ]
        },
        "subject": {
          "reference": "http://someacme.no/fhir/Patient/84c51e6c-020c-469b-b6c8-a6e9b2db6ff6"
        },
        "dosage": [
          {
            "doseAndRate": [
              {
                "doseQuantity": {
                  "value": 2,
                  "unit": "ug",
                  "system": "http://unitsofmeasure.org",
                  "code": "ug"
                }
              }
            ]
          }
        ]
      }
    },



As you can see, a FHIR json bundle will be quite lengthy and my sample file, which is a sample diagnostic report for coloscopy is right above 1000 lines and 33 kilobytes. To retrieve the medication dosage we need to go deep into the FHIR json structure sometimes and sometimes look at sibling nodes or further down. What helped me a lot while creating the extension methods I will mention next, was debugging and looking into the immediate window and inspect which kind of Entry it is. An Entry is a generic term which describes that our component is a general term in FHIR which can be many different types, such as a MedicationStatment and contain a Codable concept which ultimately will contain the dose and quantity of our medication (drug). What I did was the following to make working with the FHIR bundle in a more code friendly manner :

GENERIC APPROACH - Implement a mapping from a FHIR bundle into a DOMAIN MODEL which then can be used in your SYSTEM

  • Use the debugger and unit tests and explore in the immediate window which kind of Entry each component in the FHIR bundle is. Identity if we can cast an Entry into a correct subtype, such as a MedicationStatement. These POCO objects are available in the noted nuget packages above.
  • After finding a way to retrieve the data - generalize the retrival into extension methods, which can be chained and then make use of these extension methods into property getter logic.
  • Use a TDD approach to retrieve the data. Each property was found in the sample doc for me (about 100 properties) and I was trying to find generic ways to find these property values
  • Sometimes you need fine tuned logic too to find data. FHIR contains some extensions and different FHIR bundles, although it is a standard, may vary some.
Okay, here is all the extension methods I made for this case. I have masked the real organization names here. My approach can be used in many different scenarios for retrieving FHIR bundle data.
 
 
 
using Hl7.Fhir.Model;
using static Hl7.Fhir.Model.Observation;

namespace SomeAcme.FhirFacade.SomeProduct.HelperMethods
{

    /// <summary>
    /// Helper methods for Fhir bundle. Generic use helper methods.  
    /// </summary>
    public static class FhirHelperExtensions
    {


        public static string? SearchForPractitioner(this Bundle bundle, string procedurePath, string functionRole)
        {
            var performer = bundle.SearchForProcedure(procedurePath)?
                .Performer?.FirstOrDefault(p => p?.Function?.Coding?.FirstOrDefault()?.Display?.Equals(functionRole, StringComparison.InvariantCultureIgnoreCase) == true);
            return performer?.Function?.Coding?.FirstOrDefault()?.Code;
        }

        /// <summary>
        /// Looks up an identifier value (e.g. hpr number or similar) of practitioner
        /// </summary>
        /// <param name="practitioner"></param>
        /// <returns></returns>
        public static string? GetPractitionerIdentifierValue(this Practitioner practitioner)
        {
            return practitioner?.Identifier?.FirstOrDefault()?.Value; 
        }

        public static Organization? SearchForOrganization(this Bundle bundle, string resourcePath, bool startsWith = true)
        {
            if (startsWith)
            {
                var organizationMatching = bundle?.Entry.FirstOrDefault(e => e.FullUrl.StartsWith(resourcePath))?.Resource as Organization;
                return organizationMatching;
            }

            var organization = bundle?.FindEntry(resourcePath).FirstOrDefault()?.Resource as Organization;
            return organization; 
        }

        public static MedicationStatement? SearchMedicationStatement(this Bundle bundle, string resourcePath, bool startsWith = true)
        {
            if (startsWith)
            {
                var medicationStatementMatching = bundle?.Entry.FirstOrDefault(e => e.FullUrl.StartsWith(resourcePath))?.Resource as MedicationStatement;
                return medicationStatementMatching;
            }

            var medicationStatement = bundle?.FindEntry(resourcePath).FirstOrDefault()?.Resource as MedicationStatement;
            return medicationStatement;
        }

        public static List<MedicationStatement>? SearchMedicationStatements(this Bundle bundle, string resourcePath)
        {
            var medicationStatementsMatching = bundle?.Entry?.Where(e => e.FullUrl.StartsWith(resourcePath))?.Select(m => m.Resource)?.OfType<MedicationStatement>()?.ToList();
            return medicationStatementsMatching;
        }

        public static Dosage? GetMedicationDosageDosage(this List<MedicationStatement> medicationStatements, string displayText)
        {
            //find dosage with given display text 

            foreach (var medicationStatement in medicationStatements)
            {
                var medicationStatementMedication = medicationStatement?.Medication as CodeableConcept; 
                if (medicationStatementMedication == null)
                {
                    continue; 
                }
                var medicationCoding = medicationStatementMedication?.Coding?.FirstOrDefault(med => med.Display?.Equals(displayText, StringComparison.InvariantCultureIgnoreCase) == true);  
                if (medicationCoding != null)
                {
                    var quantity = medicationStatement?.Dosage?.FirstOrDefault();
                    return quantity; 
                }           
            }

            return null;
        }

        public static Quantity? GetMedicationDosageQuantity(this List<MedicationStatement> medicationStatements, string displayText, string? expectedUnitname = null)
        {
            //find quantity for dosage with given display text 

            foreach (var medicationStatement in medicationStatements)
            {
                var medicationStatementMedication = medicationStatement?.Medication as CodeableConcept;
                if (medicationStatementMedication == null)
                {
                    continue;
                }
                var medicationCoding = medicationStatementMedication?.Coding?.FirstOrDefault(med => med.Display?.Equals(displayText, StringComparison.InvariantCultureIgnoreCase) == true);
                if (medicationCoding != null)
                {
                    if (medicationStatement?.Dosage?.FirstOrDefault()?.DoseAndRate?.FirstOrDefault()?.Dose is Quantity quantity)
                    {
                        if (!string.IsNullOrWhiteSpace(expectedUnitname) && expectedUnitname.Equals(expectedUnitname, StringComparison.InvariantCultureIgnoreCase))
                        {
                            return quantity; 
                        }
                        return null; //found the right dosage - but the unit name does not agree 

                    }
                }
            }
            return null; 
        }


        public static string? GetOrganizationIdentifierValue(this Organization organization)
        {
            return organization?.Identifier?.FirstOrDefault()?.Value;   
        }

        public static Observation? SearchForObservation(this Bundle bundle, string resourcePath, bool startsWith = true)
        {
            if (startsWith)
            {
                var observationMatching = bundle?.Entry?.FirstOrDefault(e => e.FullUrl.StartsWith(resourcePath))?.Resource as Observation;
                return observationMatching;
            }

            var observation = bundle?.FindEntry(resourcePath).FirstOrDefault()?.Resource as Observation;
            return observation;
        }
     
        public static ComponentComponent? GetObservationComponent(this Observation observation, string observationComponentDisplayText)
        {
            //TODO : these observations is not the same as the observation in SomeProduct and must be additionally mapped (enums does not agree) 
            foreach (var observationComponent in observation.Component)
            {
                if (observationComponent?.Code?.Coding?.Any() != true)
                {
                    continue;
                }
                foreach (var observationEntry in observationComponent.Code.Coding)
                {
                    if (observationEntry?.Display.Contains(observationComponentDisplayText, StringComparison.InvariantCultureIgnoreCase) == true)
                    {
                        return observationComponent;
                    }
                }
            }
            return null;
        }

        public static string? GetObservationComponentCodeValue(this Observation observation, string observationComponentDisplayText)
        {
            //TODO : these observations is not the same as the observation in Gastronet and must be additionally mapped (enums does not agree) 
            foreach (var observationComponent in observation.Component)
            {
                if (observationComponent?.Code?.Coding?.Any() != true)
                {
                    continue; 
                }
                foreach (var observationEntry in observationComponent.Code.Coding)
                {
                    if (observationEntry?.Display.Contains(observationComponentDisplayText, StringComparison.InvariantCultureIgnoreCase) == true)
                    {
                        return observationEntry?.Code; 
                    }
                }                
            }

            return null; 
        }

        public static Patient? SearchForPatient(this Bundle bundle, string resourcePath, bool startsWith = true)
        {
            if (startsWith)
            {
                var patientMatching = bundle?.Entry?.FirstOrDefault(e => e.FullUrl.StartsWith(resourcePath))?.Resource as Patient;
                return patientMatching;
            }
            var patient = bundle?.FindEntry(resourcePath).FirstOrDefault()?.Resource as Patient;
            return patient;
        }

        public static CodeableConcept? SearchForProcedureReason(this Procedure procedure, string code)
        {
            return procedure?.ReasonCode?.FirstOrDefault(c => c?.Coding?.FirstOrDefault()?.Code?.Equals(code, StringComparison.InvariantCultureIgnoreCase) == true);
        }

        public static CodeableConcept? SearchForProcedureReasonViaDisplay(this Procedure procedure, string display)
        {
            return procedure?.ReasonCode?.FirstOrDefault(c => c?.Coding?.FirstOrDefault()?.Display?.Equals(display, StringComparison.InvariantCultureIgnoreCase) == true);
        }

        public static Procedure? SearchForProcedure(this Bundle bundle, string resourcePath, bool startsWithMatching = true)
        {
            if (startsWithMatching)
            {
                var procedureMatching = bundle?.Entry.FirstOrDefault(e => e.FullUrl.StartsWith(resourcePath))?.Resource as Procedure;
                return procedureMatching;
            }
            var procedure = bundle?.FindEntry(resourcePath).FirstOrDefault()?.Resource as Procedure;
            return procedure;
        }

        public static bool SearchForProcedureComplication(this Bundle bundle, string resourcePath, string displayText)
        {
            var procedure = SearchForProcedure(bundle, resourcePath); 
            if (procedure?.Complication?.Any() == true)
            {
                var complications = procedure.Complication.ToList();
                var complicationMatching = complications.FirstOrDefault(x => x.Coding?.FirstOrDefault()?.Display?.ToLower() == displayText);
                //TODO : consider complicaitonCode here or just mere precense ? Ask Kreftreg ? string complicationCode = complicationMatching?.Coding?.FirstOrDefault()?.Code;
                //Other developer confirmed checking mere precense is okay. going for this then.
                return complicationMatching != null;
            }
            return false; 
        }

        public static Quantity? GetObservationQuantity(this Observation observation)
        {
            var quantity = observation?.Value as Quantity;
            return quantity;             
        }

        public static DiagnosticReport? SearchForDiagnosis(this Bundle bundle, string resourcePath, bool startsWithMatching = true)
        {
            if (startsWithMatching)
            {
                var diagnosisMatching = bundle?.Entry.FirstOrDefault(e => e.FullUrl.StartsWith(resourcePath))?.Resource as DiagnosticReport;
                return diagnosisMatching;

            }
            var diagnosticReport = bundle?.FindEntry(resourcePath).FirstOrDefault()?.Resource as DiagnosticReport;
            return diagnosticReport;
        }

        public static string? GetDiagnosticProcedureCode(this DiagnosticReport diagnosticReport, int nthProcedureCode)
        {
            var codes = diagnosticReport?.Code?.Coding?.ToList();
            if (codes == null || !codes.Any() || codes.Count < nthProcedureCode+1)
            {
                return null;
            }
            var coding = codes.ElementAt(nthProcedureCode) as Coding;
            return coding.Display;
        }

        public static string? GetDiagnosisCode(this DiagnosticReport diagnostic, int position)
        {
            var diagnoses = diagnostic?.ConclusionCode?.ToList();
            if (diagnoses == null)
            {
                return null; 
            }
            if (diagnoses?.Count -1 >= position)
            {
                try
                {
                    return $"{diagnoses![position]?.Coding?.First().Code} {diagnoses[position]?.Coding.First().Display}";
                }
                catch { return null; }
            }
            return null;         
        }

        public static Extension? SearchForExtensionInsideProcedure(this Procedure procedure, string extensionUrl)
        {
            var extension = procedure?.Extension?.FirstOrDefault(e => e?.Url == extensionUrl) as Extension;
            return extension; 
        }

        public static List<Extension>? SearchForExtensionsInsideProcedure(this Procedure procedure, string extensionUrl)
        {
            var extensions = procedure?.Extension?.Where(e => e?.Url == extensionUrl)?.ToList();
            return extensions;
        }

        public static CodeableConcept? GetCodeableConceptInsideExtension(this Extension extension)
        {
            if (extension?.Value == null)
            {
                return null; 
            }
            var codeableConcept = extension?.Value as CodeableConcept;
            return codeableConcept;            
        }

        public static Extension? SearchForSubExtensionInsideExtension(this Extension extension, string extensionUrl)
        {
            var subExtension = extension?.Extension?.FirstOrDefault(e => e?.Url == extensionUrl);
            return subExtension;
        }

        public static Duration? GetExtensionDuration(this Extension extension)
        {
            var duration = extension?.Value as Duration;
            return duration; 
        }

        public static FhirBoolean? GetExtensionBoolean(this Extension extension)
        {
            return extension?.Value as FhirBoolean;
        }

        public static Coding? GetExtensionCodeValue(this Extension extension, string url, string system)
        {
            var subExtension = extension?.Extension?.FirstOrDefault(e => e?.Url == url) as Extension;
            var codeContainer = subExtension?.Value as CodeableConcept;
            return codeContainer?.Coding?.FirstOrDefault(c => c.System == system); 
        }      

        public static string? GetPatientIdentifier(this Patient patient)
        {
            return patient?.Identifier?.FirstOrDefault()?.Value;
        }


        public static string GetPatientName(this Patient patient)
        {

            var firstName = (patient?.Name?.FirstOrDefault())?.Given?.FirstOrDefault();
            var lastName = patient?.Name?.FirstOrDefault()?.Family;
            var middleName = (patient?.Name?.FirstOrDefault()?.Extension?.FirstOrDefault())?.Value?.ToString();
            return $"{firstName}{(!string.IsNullOrWhiteSpace(middleName) ? " " + middleName + " " : " ")}{lastName}";

        }

    }


}

 
 

I hope you found this article helpful in case you need to extract data from a FHIR bundle document. I am not specializing into working with FHIR, I just worked 1-2 weeks on such a FHIR bundle document and found my approach to maybe be of general interest and use. At least I found my approach
scalable for mapping each fields. Also note that I made a domain model as a model object where I put logic into the getters of a property and this corresponds to a field in the FHIR JSON bundle we want to retrieve. So I would then repeat the approach by extending the list of steps to successfully map a FHIR bundle into a domain model which THEN can be used as a better prepared model for INPUT to YOUR system. i.e. we go via a domain model that then can be input to your system where mapping will be trivial inside your system, e.g. save a domain model field into a database or similar if you want to input a FHIR json bundle and create a new POCO entity into your system and store it to a database.

GENERIC APPROACH - Implement a mapping from a FHIR bundle into a DOMAIN MODEL which then can be used in your SYSTEM

  • Use the debugger and unit tests and explore in the immediate window which kind of Entry each component in the FHIR bundle is. Identity if we can cast an Entry into a correct subtype, such as a MedicationStatement. These POCO objects are available in the noted nuget packages above.
  • After finding a way to retrieve the data - generalize the retrival into extension methods, which can be chained and then make use of these extension methods into property getter logic. These properties reside in the DOMAIN MODEL object. For example ColoscopyDomainModel .
  • Use a TDD approach to retrieve the data. Each property was found in the sample doc for me (about 100 properties) and I was trying to find generic ways to find these property values
  • Sometimes you need fine tuned logic too to find data. FHIR contains some extensions and different FHIR bundles, although it is a standard, may vary some.
  • + REMEMBER - then utilize the DOMAIN MODEL which is mapped into YOUR SYSTEM and then save the fields to a database or other storage for example. Or maybe you only want to use the domain model as is without it doing anything else than represent its data from the FHIR bundle.
Of course you will need some more infrastructure around handling FHIR documents, such as a REST API for example, but this article focused on FHIR bundle parsing. And finally, FHIR supports also the formats XML and RDF. The web site of FHIR can explain more, if you want to delve into details. I found it most helpful just to get started coding here with the GENERIC APPROACH mentioned above. HL 7 FHIR web site: http://hl7.org/fhir/

Saturday, 28 May 2022

Using expression trees to build up loops - Gauss Summation

I tested out Expression trees today in more depth and played around with Gauss summation. The Gauss summation is a well-known theorem in Calculus. It states that for a plain arithmetic sequence of numbers with a distance of 1 (i.e. 1,2,3,4,5,6...) from 1..n the sum of these numbers are equal to the formula : Sum(n) = (n*(n+1)) / 2 Johan Karl Friendrich Gauss is renowned to have come up with this as a very young student when a school teacher asked the class to sum the numbers from 1 to 100 and give him the answer. Gauss almost instantly replied '5050', which was also the correct answer. This may or may not have been the case. The formula itself can anyways be theorized by summing the largest and
smallest number and then approaching the middle of the sequence. You can add 1 and 100 to get 101, 2 and 99 to get 101 and so on. The sum is always 101 (n+1) and there are a hundred such 'pairs' (n). But we want to only sum the numbers once, so we divide by 2 => we have the Gauss summation formula ! Let's look at how to do such a summation using Expression trees in C#. But I have only created a loop algorithm here, we calculate the same answer but we instead use expression trees. In demonstrates how
we can get started with expression trees in C# using loops (it is while loop which is created here) and parameter expressions and other components, such as 'labels' used in 'gotos'. This is actually needed in
expression trees to get the looping and breaking to work. The 'SumRange' method looks like this :

public static Expression SumRange(ParameterExpression value)
{
    LabelTarget label = Expression.Label(typeof(int));

    ParameterExpression result = Expression.Variable(typeof(int), "result");
    var initializeResult = Expression.Assign(result, Expression.Constant(0));

    var innerLogicBlock = Expression.Block(
        Expression.Assign(result,
            Expression.Add(result, value)),
        Expression.PostDecrementAssign(value)
    );

    BlockExpression body = Expression.Block(
       new[] { result },
       initializeResult,
       Expression.Loop(
           Expression.IfThenElse(
            Expression.GreaterThanOrEqual(value, Expression.Constant(1)),
            innerLogicBlock,
            Expression.Break(label, result)
            ),
            label
         )
    );
    return body;
}

We pass in a parameter expression. We then declare a 'label' which is used in a 'goto' execution flow when we want to break out of our loop, created by Expression.Loop. The initializeResult is listed here inside Expression block as we want to assign the result variable to the initial value (the expression constant '0'). We then have an 'outer logic' where we have a If-Then-Else condition where we check if value is greater than or equal to 1 and then
we perform the 'inner logicl block' assigned earlier, where we assign result to itself and the value variable passed in as a parameterexpression to this method. Note, we will do some type checking via Expression.Lambda which call this SumRange method explained further below. Note the use of 'PostDecrementAssign' expression which decrements the 'value' and ensures we can exit out of the loop. It can be of course hard to follow along such expression trees without some tooling. I use the ReadableExpressions:Visualizer plugin for VS 2022 here : https://marketplace.visualstudio.com/items?itemName=vs-publisher-1232914.ReadableExpressionsVisualizers You can use it to preview expressions as shown in the below screen shot :
And our unit test passes with expected result.
 

        [Fact]
        public void SumRange()
        {
            var value = Expression.Parameter(typeof(int));
            var result = ScriptingEngine.SumRange(value);
            var expr = Expression.Lambda<Func<int, int>>(result, value);
            var func = expr.Compile(); 
             Assert.Equal(5050, func(100)); 
        }

 
As you can see, even for a simple method, we need to type a lot of code to build up an expression tree. There are helper libraries such as AgileObjects.ReadableExpressions and System.Linq.Dynamic.Core which
can help out a lot when using expression trees. Add these to your package references (.csproj) for example :

  <ItemGroup>
    <PackageReference Include="AgileObjects.ReadableExpressions" Version="3.3.0" />
    <PackageReference Include="System.Linq.Dynamic.Core" Version="1.2.18" />
  </ItemGroup>

The first package of these got a handy method ToReadableString and the last one got a handy helper class called DynamicExpressionParser, which in tandem can create pretty complex expression trees. When will you use such logic ? Most often when wanting to build up custom logic query filters. You should not allow end users to arbitrarily build up all kinds of query filters, but may offer them a set of user controls to build and combine query filters so they can retrieve data via rather complex rules. The code libs mentioned here is supported in .NET Framework 3.5 or later (and .NET Standard 1.0), so most target frameworks are supported then.

Get properties of a given type in C#

This article shows how we can find all properties with a given property type. The provided code also can find private properties or look for nullable of the property type. E.g. find all DateTime properties and also include all properties which are Nullable of DateTime, Nullable. An extension method for this looks like the following (put the method into a static class as it is an extension method) :
  

  
          /// <summary>
        /// Retrieves a list of properties (property info) with given type <paramref name="propertyType"/> in a nested object
        /// </summary>
        /// <param name="rootObject"></param>
        /// <param name="propertyType"></param>
        /// <param name="includePrivateProperties">If set to true, includes private properties</param>
        /// <param name="includeNullableVariant">If set to true, return also propertie which are the nullable variant of the <paramref name="propertyType"/>.</param>
        /// <returns>A list of properties with given <paramref name="propertyType"/>, possibly including also non-nullable variant of the type and both public and private properties set with the parameters <paramref name="includePrivateProperties"/> and <paramref name="includeNullableVariant"/></returns>
        public static IEnumerable<PropertyInfo> GetPropertiesOfType(this object rootObject, Type propertyType,
            bool includePrivateProperties = false, bool includeNullableVariant = false)
        {
            if (rootObject == null)
            {
                yield return null;
            }
            var bindingFlagsFilter = !includePrivateProperties ? BindingFlags.Public | BindingFlags.Instance : BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance;
            var propertiesOfType = rootObject.GetType().GetProperties(bindingFlagsFilter)
                .Where(p => p.PropertyType == propertyType || (includeNullableVariant && propertyType == Nullable.GetUnderlyingType(p.PropertyType)))
                .ToList();
            foreach (var prop in propertiesOfType)
            {
                yield return prop;
            }
            var nestableProperties = rootObject.GetType().GetProperties(bindingFlagsFilter)
              .Where(p => p.PropertyType.IsClass && p.PropertyType != typeof(string))
              .ToList(); //ignoring properties of type strings as they are not nested, though a class
            foreach (var prop in nestableProperties)
            {
                if (prop.GetIndexParameters().Length > 0)
                {
                    continue; //skip indexer properties 
                }
                var rootObjectLevel = prop.GetValue(rootObject, null);
                if (rootObjectLevel == null)
                {
                    continue;
                }
                foreach (var propertyAtLevel in GetPropertiesOfType(rootObjectLevel, propertyType, includePrivateProperties, includeNullableVariant))
                {
                    yield return propertyAtLevel;
                }
            }
        }

We find the properties matching the property type and then recursively fetch such properties at nested levels too if the property is a class and therefore can contain sub properties. We end up with all the properties of a given type. As we see, we adjust the binding flags to include private properties too or not. And we use the Nullable.GetUnderlyingType method to match the underlying type in case we want to look for DateTime and DateTime? properties. This method is fairly fast, in the order of a few milliseconds (1-5 when I tested for an ordinary two level nested object.) But we are using reflection here and the method could be faster if we made use of some other techniques, perhaps with
IL 'magic'. I have not found a way to do this yet though.. Here is another utility method (extension method) for finding 'property paths'. This is handly if you want to craft an SQL select statement for example as we need the fully qualified path perhaps if our tooling creates fields in the database similar to POCO object
and the nested object is similar to table structure. Maybe your nested properties are mapped to SomeInnerTable1_SomeField1 and so on. Anyways, it is handly to have 'property paths' to the properties to get a fast overview of where the properties are located in the nested structure of your (possibly complex) object.



         /// <summary>
        /// This method looks for properties of given type in a nested object (e.g. a form data contract) 
        /// </summary>
        /// <param name="rootObject"></param>
        /// <param name="propertyType"></param>
        /// <returns></returns>
        private IEnumerable<string> GetPropertyPathsForType(object rootObject, Type propertyType, string prefixAtLevel = "")
        {
            if (rootObject == null)
            {
                yield return string.Empty;
            }
            var propertiesOfType = rootObject.GetType().GetProperties(BindingFlags.Public | BindingFlags.Instance)
                .Where(p => p.PropertyType == propertyType)
                .ToList();

            foreach (var prop in propertiesOfType)
            {
                if (string.IsNullOrWhiteSpace(prefixAtLevel))
                {
                    yield return prop.Name; //root properties have no prefix 
                }
                else
                {
                    yield return prefixAtLevel.TrimStart('.') + "." + prop.Name;
                }
            }

            var nestableProperties = rootObject.GetType().GetProperties(BindingFlags.Public | BindingFlags.Instance)
              .Where(p => p.PropertyType.IsClass && p.PropertyType != typeof(string))
              .ToList(); //ignoring properties of type strings as they are not nested, though a class

            foreach (var prop in nestableProperties)
            {
                if (prop.GetIndexParameters().Length > 0)
                {
                    continue; //skip indexer properties - this is identified as required 
                }
                var rootObjectLevel = prop.GetValue(rootObject, null);
                if (rootObjectLevel == null)
                {
                    continue;
                }
                foreach (var propertyAtLevel in GetPropertyPathsForType(rootObjectLevel, propertyType, prefixAtLevel + "." + prop.Name))
                {
                    yield return propertyAtLevel.TrimStart('.').TrimEnd('.');
                }
            }
        }


The code above could use the first method more to support including public and private properties, but I leave it out 'as an exercise to the reader' as text books often states. So this code is very handy if you for example at work need to find 'all the datetime properties in a domain object' and similar cases. Maybe you want to deny all datetimes have a future datetime in case it is a report for a patient treatment report being performed yesterday and so on, and for
that particular model, there will be no future date time values.

Wednesday, 6 April 2022

Abstract factory - registering factory methods

Today, instead of relying on factories we often resort to dependency injection. However, you can 'blueprint' objects by offering a helper class to register both interfaces with concrete classes and default parameters if you want to instantiate a type of object with given parameters using an abstract factory class The following code shows how we can start building such a registration approach, supporting also unregistering factories and creating objects which are registered. You will have to add thread safety and better checking if a given type T and parameters P1..PN are registered, here is though a simple Abstract factory. Paste the code into Linqpad 6 or Linqpad 7 to test it out !
 
 
 void Main()
{

 var factory = new AbstractFactory(); 
 
 factory.Register<IDrive>(() => new FoulwheeledDrive());
 factory.Register<Car, string, string, IDrive>((string make, string model, IDrive drive) => new Car(make, model, drive)); 
 
 var car = factory.Create<Car, string, string, IDrive>("BMW", "M5", factory.Create<IDrive>());
 
 car.ToString().Dump();
 
 //can also unregister a factory method
 //factory.Unregister<Car, string, string, IDrive>();
 //var car2 = factory.Create<Car, string, string, IDrive>("BMW", "M5", factory.Create<IDrive>());
 
	
}

public interface IVehicle {
 string Model { get; set; }
 string Make { get; set; }
}

public interface IDrive {
 string Drive();
}

public class FoulwheeledDrive : IDrive {
 public string Drive() => "Car driving with four wheeled motion";
}

public class Car  {

  public string Make { get; set; }  
  public string Model { get; set; }
  
  public IDrive Drive { get; set; }
 
  public Car(string model, string make, IDrive drive){
    Model = model; 
 	Make = make;
	Drive = drive;
  }
  
  public override string ToString() => $"{Model} {Make} : {Drive.Drive()}";
 
}

public class AbstractFactory {

 private readonly Dictionary<Type, Delegate> _factories = new (); 
 
 
 public void Register<T>(Func<T> factory) => _factories[typeof(Func<T>)] = factory;
 public void Register<T,P>(Func<P,T> factory) => _factories[typeof(Func<P,T>)] = factory;
 public void Register<T,P1, P2>(Func<P1, P2, T> factory) => _factories[typeof(Func<P1, P2, T>)] = factory;
 public void Register<T, P1, P2, P3>(Func<P1, P2, P3, T> factory) => _factories[typeof(Func<P1, P2, P3, T>)] = factory;
 public void Register<T, P1, P2, P3, P4>(Func<T> factory) => _factories[typeof(Func<P1, P2, P3, P4, T>)] = factory;
 public void Register<T, P1, P2, P3, P4, P5>(Func<T> factory) => _factories[typeof(Func<P1, P2, P3, P4, P5, T>)] = factory;
 
 public void Unregister<T>() => _factories.Remove(typeof(Func<T>)); 
 public void Unregister<T,P>() => _factories.Remove(typeof(Func<P,T>));
 public void Unregister<T,P1, P2>() => _factories.Remove(typeof(Func<P1, P2, T>));
 public void Unregister<T,P1, P2, P3>() => _factories.Remove(typeof(Func<P1, P2, P3, T>)); 
 public void Unregister<T,P1, P2, P3, P4>() => _factories.Remove(typeof(Func<P1, P2, P3, P4, T>));
 public void Unregister<T,P1, P2, P3, P4, P5>() => _factories.Remove(typeof(Func<P1, P2, P3, P4, P5, T>));
 
 public T Create<T>() => ((Func<T>)_factories[typeof(Func<T>)])();
 public T Create<T, P>(P p) => ((Func<P,T>)_factories[typeof(Func<P,T>)])(p);
 public T Create<T, P1, P2>(P1 p1, P2 p2) => ((Func<P1, P2,T>)_factories[typeof(Func<P1, P2,T>)])(p1, p2);
 public T Create<T, P1, P2, P3>(P1 p1, P2 p2, P3 p3) => ((Func<P1, P2, P3, T>)_factories[typeof(Func<P1, P2, P3, T>)])(p1, p2, p3);
 public T Create<T, P1, P2, P3, P4>(P1 p1, P2 p2, P3 p3, P4 p4) => ((Func<P1, P2, P3, P4, T>)_factories[typeof(Func<P1, P2, P3, P4, T>)])(p1, p2, p3, p4);
 public T Create<T, P1, P2, P3, P4, P5>(P1 p1, P2 p2, P3 p3, P4 p4, P5 p5) => ((Func<P1, P2, P3, P4, P5, T>)_factories[typeof(Func<P1, P2, P3, T>)])(p1, p2, p3, p4, p5);
 
}


 
 

The registration of the factories then instantiate the instance. Note that we do not use the new operator when instantiating the object. We avoid using Activator.CreateInstance and reflection using this approach. Why use this pattern when we want to create 'blueprint' objects and want to have standard ways of instantiating them and for example avoid using concrete types. Most IoC frameworks allow you to create compound classes and supply parameters, so the abstract factory pattern seems a bit 'crude', but it is used many places still.

Friday, 1 April 2022

GraphQL in Asp.Net Core - Creating a flexible API

More and more .NET Developers have heard about GraphQL. This started as an in-house project in Facebook 2012 to provide a flexible way of sending customized data to mobile clients. Giving the clients the possible to query after tailored data meant sending less data over the wire to the mobiles with less bandwidth. As cell phones moves over to 5G networks, the issue means less and less (in urban areas with good base station coverage), however we should of course seek to always optimize our data transfer as pure bandwidth usage is always a valued thing to optimize. And added dimension is the less cost of creating APIs as we can tailor our data needs. Instead of creating methods for either returning lookup ids and then querying after entire data objects, we can project only the data we need to retrieve to present data on the mobile clients in a meaningful way. Whatever makes your boat rock for showing interest in GraphQL, this article will discuss how you can get started with GraphQL in Asp.Net Core. I have prepared a demo here:
 
  https://github.com/toreaurstadboss/AspNetCore-GraphQLDemo
 
The demo repository shows a list of the tallest mountains in the municipialites in Norway. Norway is a land of mountains and it is always to know which mountain is the very tallest in the municipiality you are visiting! (I enjoy mountain climbing and hiking now and then in my spare time). The demo page shows a text area where you can customize the data to load here. Of course we can only load the data provided for us. We can also use the Ui playground for GraphQL added for us here too:
First off, we need to grab some Nuget packages for GraphQL. We will be using Asp.Net Core 3.1. in this article.
 
        <PackageReference Include="GraphQL" Version="2.4.0" />
	<PackageReference Include="GraphQL.Server.Transports.AspNetCore" Version="3.4.0" />
	<PackageReference Include="GraphQL.Server.Transports.WebSockets" Version="3.4.0" />
	<PackageReference Include="GraphQL.Server.Ui.Playground" Version="3.5.0-alpha0046" />  
 
Then we need to specify in our Startup class the needed setup.
 
Startup.cs
using AspNetCore_GraphQLDemo.GraphQL; using AspNetCore_GraphQLDemo.GraphQL.Messaging; using Data; using Data.Repositories; using GraphQL; using GraphQL.Server; using GraphQL.Server.Ui.Playground; using Microsoft.AspNetCore.Builder; using Microsoft.AspNetCore.Diagnostics; using Microsoft.AspNetCore.Hosting; using Microsoft.AspNetCore.Http; using Microsoft.AspNetCore.WebSockets; using Microsoft.EntityFrameworkCore; using Microsoft.Extensions.Configuration; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Hosting; using Newtonsoft.Json; namespace AspNetCore_GraphQLDemo { public class Startup { private readonly IWebHostEnvironment _env; public Startup(IConfiguration configuration, IWebHostEnvironment env) { _env = env; Configuration = configuration; } public IConfiguration Configuration { get; } // This method gets called by the runtime. Use this method to add services to the container. public void ConfigureServices(IServiceCollection services) { // If using IIS: services.Configure<IISServerOptions>(options => { options.AllowSynchronousIO = true; }); services.AddControllersWithViews(); services.AddHttpContextAccessor(); services.AddRazorPages().AddRazorRuntimeCompilation(); services.AddDbContext<MountainDbContext>(options => { options.UseSqlServer(Configuration.GetConnectionString("DefaultConnection")); }); services.AddScoped<IMountainRepository, MountainRepository>(); services.AddScoped<IDependencyResolver>(s => new FuncDependencyResolver(s.GetRequiredService)); services.AddScoped<MountainSchema>(); services.AddSingleton<MountainMessageService>(); services.AddSingleton<MountainDetailsDisplayedMessageService>(); services.AddGraphQL(x => { x.EnableMetrics = true; x.ExposeExceptions = _env.IsDevelopment(); x.SetFieldMiddleware = true; }).AddGraphTypes(ServiceLifetime.Scoped) .AddUserContextBuilder(httpContext => httpContext.User) .AddDataLoader() .AddWebSockets(); services.AddCors(options => { options.AddPolicy(name: "MyAllowSpecificOrigins", builder => { builder.AllowAnyOrigin().AllowAnyMethod(); }); }); } //static IEnumerable<Type> GetGraphQlTypes() //{ // return typeof(Startup).Assembly // .GetTypes() // .Where(x => !x.IsAbstract && // (typeof(IObjectGraphType).IsAssignableFrom(x) || // typeof(IInputObjectGraphType).IsAssignableFrom(x))); //} // This method gets called by the runtime. Use this method to configure the HTTP request pipeline. public void Configure(IApplicationBuilder app, IWebHostEnvironment env) { if (env.IsDevelopment()) { app.UseDeveloperExceptionPage(); app.UseBrowserLink(); } app.UseExceptionHandler(errorApp => { errorApp.Run(async context => { context.Response.Redirect("/Error"); context.Response.StatusCode = 500; var exceptionHandlerPathFeature = context.Features.Get<IExceptionHandlerPathFeature>(); var exception = exceptionHandlerPathFeature.Error; var result = JsonConvert.SerializeObject(new { error = exception.Message }); context.Response.ContentType = "application/json"; await context.Response.WriteAsync(result); }); }); app.UseStaticFiles(); app.UseRouting(); app.UseCors("MyAllowSpecificOrigins"); app.UseWebSockets(); app.UseGraphQLWebSockets<MountainSchema>("/graphql"); //app.UseAuthorization(); app.UseEndpoints(endpoints => { endpoints.MapDefaultControllerRoute(); }); app.UseGraphQL<MountainSchema>(); if (env.IsDevelopment()) { app.UseGraphQLPlayground(new GraphQLPlaygroundOptions { }); } } } }
In ConfigureServices method above we register the schema for our GraphQL method.
 
 
    services.AddScoped<MountainSchema>(); 
 
 
We also add GraphQL itself and setup also web sockets (which are needed for GraphQL).
 
       services.AddGraphQL(x =>
                {
                    x.EnableMetrics = true; x.ExposeExceptions = _env.IsDevelopment(); x.SetFieldMiddleware = true; }).AddGraphTypes(ServiceLifetime.Scoped)
            .AddUserContextBuilder(httpContext => httpContext.User)
            .AddDataLoader()
            .AddWebSockets(); 
 
Just as a side note, you want to add Cors also:
 
     services.AddCors(options =>
            {
                options.AddPolicy(name: "MyAllowSpecificOrigins",
                    builder =>
                    {
                        builder.AllowAnyOrigin().AllowAnyMethod();
                    });
            });
 
Inside Configure method we add also the following to enable GraphQL:
 
           app.UseCors("MyAllowSpecificOrigins");

            app.UseWebSockets();

            app.UseGraphQLWebSockets<MountainSchema>("/graphql");

            //app.UseAuthorization();

            app.UseEndpoints(endpoints =>
            {
                endpoints.MapDefaultControllerRoute();
            });

            app.UseGraphQL<MountainSchema>();
            if (env.IsDevelopment())
            {
                app.UseGraphQLPlayground(new GraphQLPlaygroundOptions
                {
                    
                });
            }
        }  
 
Our Mountainschema looks like this:
MountainSchema.cs
using AspNetCore_GraphQLDemo.GraphQL.Types; using AspNetCore_GraphQLDemo.GraphQL.Types.Directives; using GraphQL; using GraphQL.Instrumentation; using GraphQL.Types; namespace AspNetCore_GraphQLDemo.GraphQL { public class MountainSchema : Schema { public MountainSchema(IDependencyResolver resolver) : base(resolver) { Query = resolver.Resolve<MountainQuery>(); Mutation = resolver.Resolve<MountainMutation>(); Subscription = resolver.Resolve<MountainSubscription>(); RegisterDirective(new LowercaseDirective()); RegisterDirective(new OrderbyDirective()); var builder = new FieldMiddlewareBuilder(); builder.Use<LowercaseFieldsMiddleware>(); builder.ApplyTo(this); builder.Use(next => { return context => { return next(context).ContinueWith(x => { var c = context; var result = x.Result; result = OrderbyQuery.OrderIfNecessary(context, result); return result; }); }; }); builder.ApplyTo(this); //builder.Use<CustomGraphQlExecutor<MountainSchema>>(); //builder.ApplyTo(this); } } }
We pass in a IDependencyResolver (dependency!) into the constructor and resolve the classes we desire (we inherit from Schema class). We wire up our schema here to the Query, Mutation and Subscription we desire and register directives. Here is how the Query property is set:
 
MountainQuery.cs
using AspNetCore_GraphQLDemo.GraphQL.Types; using Data; using Data.Repositories; using GraphQL.Types; namespace AspNetCore_GraphQLDemo.GraphQL { public class MountainQuery : ObjectGraphType { public MountainQuery(IMountainRepository mountainRepository) { Field<ListGraphType<MountainType>>("mountains", resolve: context => mountainRepository.GetAll() ); FieldAsync<MountainType>("mountain", arguments: new QueryArguments(new QueryArgument<NonNullGraphType<MountainIdInputType>> {Name = "id"}), resolve: async context => { var mountain = context.GetArgument<MountainInfo>("id"); var mountainFromDb = await mountainRepository.GetById(mountain.Id); return mountainFromDb; }); //FieldAsync<MountainType>("selectmountain", // arguments: new QueryArguments(new QueryArgument(typeof(int)) { Name = "id" }), // resolve: async context => // { // var mountain = context.GetArgument<MountainInfo>("id"); // var mountainFromDb = await mountainRepository.GetById(mountain.Id); // return mountainFromDb; // }); //sadly, we need to inherit from IGraphType and cannot just have simple scalar arguments in GraphQL.Net.. } } }
As you can see, we can define multiple queries. We inherit from ObjectGraphType and pass in a IMountainRepository. This is an interface for your repository, which fetches data via Entity Framework Core and you can then load data into GraphQL from the local database (The DEMO uses Sql Server (SQLEXPRESS)) via EF Core in a simple manner by only providing the repo via dependency injection. We define via the methods Field and FieldAsync our methods (note the use of string constants as a string value we can use in GraphQL queries of ours that resides in the Schema) and the resolve lambda tells how data is to be fetched. We can specify arguments also. The "mountain" FieldAsync method also accepts arguments via the
arguments lambda and this allows us parameterized access to our data. Over to the Subscription property. It looks like this:
 
using AspNetCore_GraphQLDemo.GraphQL.Messaging;
using AspNetCore_GraphQLDemo.GraphQL.Types;
using GraphQL.Resolvers;
using GraphQL.Types;

namespace AspNetCore_GraphQLDemo.GraphQL
{
    public class MountainSubscription : ObjectGraphType
    {
        public MountainSubscription(MountainDetailsDisplayedMessageService mountainDetailsDisplayedMessageService)
        {
            Name = "Subscription";
            AddField(new EventStreamFieldType
            {
                Name = "detailsDisplayed",
                Type = typeof(MountainDetailsMessageType),
                Resolver = new FuncFieldResolver<MountainDetailsMessage>(c => c.Source as MountainDetailsMessage),
                Subscriber = new EventStreamResolver<MountainDetailsMessage>(c => mountainDetailsDisplayedMessageService.GetMessages())
            });
        }
    }
}
 
 
Here we inherit from ObjectGraphType (as we did for Query) and we use the MountainDetailsDisplayedMessageService. This was added as a (concrete class) singleton in the Startup.cs file. The message service uses RxJs serverside to handle the Pub-sub pattern of the subscriber. We are using System.Reactive.Subjects here.
 
MountainSubscription.cs
using System; using System.Reactive.Linq; using System.Reactive.Subjects; namespace AspNetCore_GraphQLDemo.GraphQL.Messaging { public class MountainDetailsDisplayedMessageService { private readonly ISubject<MountainDetailsMessage> _messageStream = new ReplaySubject<MountainDetailsMessage>(1); public MountainDetailsMessage AddMountainDetailsMessage(int id) { var message = new MountainDetailsMessage { Id = id }; _messageStream.OnNext(message); return message; } public IObservable<MountainDetailsMessage> GetMessages() { return _messageStream.AsObservable(); } } }
The mutation looks like this:
MountainMutation.cs
using AspNetCore_GraphQLDemo.GraphQL.Messaging; using AspNetCore_GraphQLDemo.GraphQL.Types; using Data; using Data.Repositories; using GraphQL.Types; namespace AspNetCore_GraphQLDemo.GraphQL { public class MountainMutation : ObjectGraphType { public MountainMutation(IMountainRepository mountainRepository, MountainMessageService mountainMessageService) { FieldAsync<MountainType>("createMountain", arguments: new QueryArguments( new QueryArgument<NonNullGraphType<MountainInputType>> {Name = "mountain"}), resolve: async context => { var mountain = context.GetArgument<MountainInfo>("mountain"); await mountainRepository.AddMountain(mountain); mountainMessageService.AddMountainAddedMessage(mountain); return mountain; }); FieldAsync<MountainType>("removeMountain", arguments: new QueryArguments( new QueryArgument<NonNullGraphType<MountainIdInputType>> { Name = "id" }), resolve: async context => { var mountain = context.GetArgument<MountainInfo>("id"); await mountainRepository.RemoveMountain(mountain.Id); return mountain; }); } } }
We can create a mountain like this in GraphQL Query:
 
 mutation {
  createMountain(mountain: {
   county: "Svalbard"
  muncipiality: "Svalbard"
  officialName: "Newtontoppen"
  referencePoint: "Isbjønn på toppen"
  comments: "Husk rask snøskuter",
  metresAboveSeaLevel: "1713",
  primaryFactor: "1713"
  }) {    
    id
  }
} 
  
 
And we can remove a mountain (don't we all?) like this:
 

# Write your query or mutation here
mutation {
  removeMountain(id: {
    id: 370
  }) { id }
}
  
 
If you clone the repo you will find more source code concerning directives such as lowercase and sorting. As you saw in MountainSchema I use the FieldMiddlewareBuilder to do the sorting as this needs to tap into the pipeline more of GraphQL.Net. We also need some more code - for the client side of course. The client side code relies on Apollo Client lib like this:
 
index.cshtml
<script src="https://unpkg.com/apollo-client-browser@1.7.0"></script>
The libman.json file (the similar file to package.json when it comes to specifying client-side libraries in .net core mvc solutions) of the demo solution looks like this I have used looks like this:
 
libman.json
{ "version": "1.0", "defaultProvider": "cdnjs", "libraries": [ { "library": "twitter-bootstrap@4.2.1", "destination": "wwwroot/lib/bootstrap", "files": [ "js/bootstrap.bundle.js", "css/bootstrap.min.css" ] }, { "library": "jquery@3.3.1", "destination": "wwwroot/lib/jquery", "files": [ "jquery.min.js" ] }, { "provider": "unpkg", "library": "font-awesome@4.7.0", "destination": "wwwroot/lib/font-awesome/" }, { "provider": "unpkg", "library": "toastr@2.1.4", "destination": "wwwroot/lib/toastr/" } ] }
We then need some client side code to load data from GraphQL server of ours.
 
  <script>

    function LoadGraphQLDataIntoUi(result) {

        var tableBody = $("#mountainsTableBody");
        tableBody.empty();

        var tableHeaderRow = $("#mountainsTableHeaderRow");
        tableHeaderRow.empty();

        var rowIndex = 0;

        result.data.mountains.forEach(mountain => {

            if (rowIndex == 0) {
                Object.keys(mountain).forEach(key => {
                    if (key === '__typename') {
                        return;
                    }
                    tableHeaderRow.append(`<th>${key}</th>`);
                });;
            }

            tableBody.append('<tr>');

            Object.keys(mountain).forEach(key => {
                if (key === '__typename') {
                    return;
                }
                if (key === 'id') {
                    tableBody.append(`<td><a href='/home/mountaindetails/?id=${mountain[key]}'><i class='fa fa-arrow-right'></i></a> ${mountain[key]}</td>`);
                    return;
                }
                tableBody.append(`<td>${mountain[key]}</td>`);

            });

            tableBody.append('</tr>');

            rowIndex++;

        });

        toastr.success('Loaded GraphQL data from server into the UI successfully.');


    }

    $("#btnConnect").click(function () {
        ConnectDemo();

    });


    $("#btnLoadData").click(function () {
        var gqlQueryContents = $("#GraphQLQuery").val();
        LoadGraphQLData(gqlQueryContents, LoadGraphQLDataIntoUi);
        toastr.info('Retrieving data from API using GraphQL.');
    });

    $(document).ready(function () {

        console.log('loading');

        var initialQuery = `
                {
                    mountains {
                        id
                        fylke: county
                        kommune: muncipiality
                        hoydeOverHavet: calculatedMetresAboveSeaLevel
                        offisieltNavn: officialName
                        primaerfaktor: calculatedPrimaryFactor
                        referansePunkt: referencePoint
                    }
                }`;

        $("#GraphQLQuery").val(initialQuery);

    });

</script>
 
 
And then a method using Apollo client lib to load the data:
 
 /**
 * Loads GraphQL data specified by query expression and passes the 'result' array to the callBackFunction
 * callBackFunction should be Js method (function) that accept one parameter, preferably called result, which is an object
 * that contains a result.data object.
 */
function LoadGraphQLData(gqlQuery, callBackFunction) {

    var apolloClient = new Apollo.lib.ApolloClient({
        networkInterface: Apollo.lib.createNetworkInterface({
            uri: 'http://localhost:2542/graphql',
            transportBatching: true,
        }), connectToDevTools: true
    });
    var query = Apollo.gql(gqlQuery);

    apolloClient.query({
        query: query,
        variables: {}
    }).then(result => {
        callBackFunction(result);
    }).catch(error => {
        //debugger
        toastr.error(error, 'GraphQL loading failed');
    });
}
 

Saturday, 19 March 2022

Using C# 9 language features in .NET Framework and .NET Standard projects

C# 7.0 came out in March 2017 and Microsoft has published other frameworks later, such as .NET Core and .NET 5 plus .NET 6. If you are working with a .NET Framework based solution (or .NET Standard 2.0), you can actually get support for C# 8 and C# 9 language version - enabling you to utilize more of C# language features. The following steps can be used to enable C# language 9 in for example .NET Framework 4.8 (tested and verified that I could use records (a C# 9 language feature).
  • Specify in the .csproj file(s) that you want to use <LangVersion> element set to 9.0
  • Consider using a file called Directory.Build.props (at the root level of your solution) (Case sensitive on Linux) with this shared setting to enabled C# 9.0 version in all projects.
  • Using C# 9 language version also requires you to include a small file in each project listed below, call it IsExternalInitPatch.cs for example.
File IsExternalInitPatch.cs should include this :

  
namespace System.Runtime.CompilerServices
{
    internal static class IsExternalInit { }
}
  


Now you can start playing around with C# 9 in a .NET Framework 4.8 solution for example, which earlier has been limited to C# 7.1 and no later language version features of C#.

namespace SomeAcme.SomeProduct.Common.Test
{
    /// <summary>
    /// This is just a test of csharp 9 for SomeAcme.SomeProduct
    /// Note that Directory.Build.props in this branch uses LangVersion set to 9.0 and we need the file IsExternalInit.cs in every project
    /// </summary>
    /// <remarks>
    /// See these two urls: 
    /// https://btburnett.com/csharp/2020/12/11/csharp-9-records-and-init-only-setters-without-dotnet5.html
    /// https://blog.ndepend.com/using-c9-record-and-init-property-in-your-net-framework-4-x-net-standard-and-net-core-projects/
    /// </remarks>
    [TestFixture]
    public class TestOutCsharpNine
    {

        public record Operasjon (DateTime StartTid, bool ErElektiv, string PasientNavn);


        [Test]
        public void Test_Records_ChsharpNine_And_Deconstruction_And_Discardable_Variables()
        {
            var op = new Operasjon(DateTime.Today.AddHours(8).AddMinutes(15), true, "Bjarne Brøndbo");
            (_, _, string pasientNavn) = op;
            pasientNavn.Should().Be(op.PasientNavn);
        }

        [Test]
        public void Test_Init_Only_Props()
        {
            var op = new OperasjonWithInitOnlyProps
            {
                ErElektiv = true,
                PasientNavn = "Thomas Brøndbo"
            };
            // op.PasientNavn = "foo"; 
            //uncommenting line above should demonstrate init only property giving compiller error if trying to mutate or alter this property
            op.PasientNavn.Should().Contain("Brøndbo");
        }

        [DataContract]
        public class OperasjonWithInitOnlyProps
        {
            [DataMember]
            public string PasientNavn { get; init; }
            [DataMember]
            public bool ErElektiv { get; init; }
        }
    }
}


The CSharp compiler sets up default the CSharp language features according to these rules: The compiler determines a default based on these rules:
Target framework	version	C# language version default
.NET	6.x	C# 10
.NET	5.x	C# 9.0
.NET Core	3.x	C# 8.0
.NET Core	2.x	C# 7.3
.NET Standard	2.1	C# 8.0
.NET Standard	2.0	C# 7.3
.NET Standard	1.x	C# 7.3
.NET Framework	all	C# 7.3
So .NET Framework and .NET Standard based solution has not gotten per default any modernization of C# sharp features since March 2017 (five years ago), but we can with some small modification still use C# 9.0 which came out 1.5 years ago. Of course, this C# language version is meant to be used with .NET 5, so do not expect everything to be supported on it. However, chances are high that much of C# 8 and C# 9 language features could be handy to use in many .NET Framework and .NET Standard based projects. For example, records with their support for immutability is definately a big new thing in C# compared to what is avilable in C# language version 8 or earlier. Lastly, you must also consider how to build C# 9 language features (which assumes .NET 5 available) on a build server. For Team City for example, you must install .NET 5 SDK on the build agent.
Also, most likely you have for example a MS Build step in Team City, so you should use MS Build 16 (VS 2019 Build tools) and install the build tools for VS 2019 on the build agent from this url or google for Build Tools for VS 2019: https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=BuildTools&rel=16&src=myvs&utm_medium=microsoft&utm_source=my.visualstudio.com&utm_campaign=download&utm_content=vs+buildtools+2019 For Azure Devops, choose the VS 2022 agent. I still had to add a "Use NET Core" task and choose 'Package to install' set to 'SDK (contains runtime)', the YAML looks like this:

steps:
- task: UseDotNet@2
  displayName: 'Use .NET Core sdk 5.0.100'
  inputs:
    version: 5.0.100
    includePreviewVersions: true

Also note this - albeit you might have .NET Framework 4.8 in a project, your config file like app.config might have this :
 
  


<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <startup> 
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.6.1" />
    </startup>
	<appSettings>
 
 
The supportedRuntime might force you to in a specific project to have a LangVersion set to a lower value anyways. So you might need for example to have LangVersion set to 7.1 in one project and default to LangVersion 9.0. To sum up :
  • .NET Framework and .NET Standard can still use C# language version 8 or 9. You need to do the adjustments I mentioned in this article.
  • C# language version 10 is only supported by .NET 6. To use this language version you have to upgrade framework..
  • Also - test out new language features in one project first and use the basic features first. If you use advanced language features of C# language version 8 or 9 you might consider some glitches.. However, you should get a compiler warning for most errors you encounter.
  • Don't forget that your build agent must be able to build the solution too. So you can use VS 2022 hosted agent and consider also the USE NET Core Sdk task I mentioned here if you build in Azure Devops. If you use a self-hosted agent, like Team City on-premises build agent, you need to install the newest VS 2019 SDK / Build Tools to ensure that you have the C# langversion.
In the Developer command prompt on the build agent you can run this command
 
  csc -langversion:? 
 
This should output the langversions of C# your build agent supports. It also works on a developer PC (use the VS 2019 command prompt). As I noted, C# 10 is only supported in .NET 6. We might have a future situation where C# 11 is still supported in a .NET 6 solution - I am not sure what Microsoft is planning here. But for other and earlier frameworks, it looks like C# 9 is the end of the road of language versions - we have to upgrade to .NET 6 to utilize newer language features (or consider dragging in Nuget compiler packages ..)

Saturday, 12 March 2022

Added NinUtilsNorway - shows modulo 11 algorithm for verifying Norwegian personal identifier numbers

I added a library to Nuget called NinUtilsNorway. Nin is an acronym for Norwegian Identifier Number. It is handy for verifying that Norwegian personal identifiers - or PID - are correct. It supports addional formats of Personal Identifiers Numbers (fnr - fødselsnummere) and these kinds of PID / NIN are supported:
  • Fnr (ordinary PIDs / NINs)
  • D-number - handed out to those working in Norway in a temporary period of some months or years - 'guest workers'
  • H-number - "Nødnummer" - given to tourists, unidentified people et cetera - those on temporary visit to Norway on visa e.g
  • DUF-number - given to asylum seekers by UDI
  • FH-number - a variant of H-number with relaxed formatting - supports more numbers
Install it using these commands : .NET Framework 4.6.2 - at least .NET Framework 4.7.1 recommended : Install-Package NinUtilsNorway -Version 1.1.0 .NET Core and .NET 5 and .NET 6 : dotnet add package NinUtilsNorway --version 1.1.0 In the future, in 2032, a new standard - also called PID - will replace todays format. New citizens (newborns etc) will get new NIN / PID fødselsnummere. But people born before 2032 will default keep their
NIN / PID /fnr. We can still read out age and gender from these fnr using the existing modulo-11 algorithm - actually both the age and the gender are readily resolved without checking the two last controls digits via a modulo 11 algorithm. They follow rules establised. You can browse the source code on my Github repo to see how we resolve information from NIN / PID. The Github repo is here: Github repo Here you can browse the source code of NinUtils. The lib is written in netstandard 2.0 so you can use .NET Framework 4.7+ (theoretically 4.6.2 can also be supported) and .NET Core or .NET 5 and .NET 6. A sample client in .NET 6 Console app is:
 
 
using static NinUtilsNorway.NinUtilsNorway;


// See https://aka.ms/new-console-template for more information
Console.WriteLine("Hello, World!");

Console.WriteLine("Enter your fnr: ");
string fnr =  Console.ReadLine();

bool isValid = IsValidNin(fnr);
NinUtilsNorway.Gender gender = GetGender(fnr);

Console.WriteLine($"The fnr {fnr} is valid? {isValid} Gender of fnr is: {gender}");

 
This page shows information how the last two numbers of a fnr is used to verify the validity of a fnr: http://www.fnrinfo.no/Teknisk/KontrollsifferSjekk.aspx In short we calculate the last two control digits k1 and k2 and check that both numbers are divisible by 11 like this - C# code: Note - for a fnr with digits d1, d2, .. d11 we calculate mathematically the sum of each digit by multiplying these weights: k1 : weights are {3,7,6,1,8,9,4,5,2,1} k2 : weights are { 5, 4, 3, 2, 7, 6, 5, 4, 3, 2, 1 } Check the link I gave above, it has a very easy example. The modulo-11 algorithm is very similar to what is used in Norwegian banks for KID - customer identifier in checques / billings to have another example.
 
 

   /// >summary<
        /// Calculates validity of Nin according to modulo 11 algorithm. 
        /// >/summary<
        /// >param name="nin"<>/param<
        /// >returns<>/returns<
        /// >remarks<>see href="http://www.fnrinfo.no/Teknisk/KontrollsifferSjekk.aspx"
        /// Example of a Modulo-11 algorithm mathematical basis is shown here: 
        /// >see href="http://www.pgrocer.net/Cis51/mod11.html"/<
        /// >/remarks<
        public static bool IsValidNin(string nin)
        {
            nin = nin?.Trim();
            if (nin?.Length != 11)
            {
                return false;
            }

            if (!long.TryParse(nin, out var _))
            {
                return false;
            }

            if (IsDNumber(nin))
            {
                nin = (byte.Parse(nin[0].ToString()) - 4).ToString() + nin.Skip(1);
            }

            int k1 = 0, k2 = 0; //weighted sums be 
            int[] k1_weights = new int[] { 3, 7, 6, 1, 8, 9, 4, 5, 2, 1 };
            foreach (var item in nin.Select((digit, index) =< (digit, index)))
            {
                if (item.index == 10)
                {
                    break; //only considering first 10 digits of nin
                }
                k1 += int.Parse(item.digit.ToString()) * k1_weights[item.index];
            }
            if (k1 % 11 != 0)
            {
                return false; //k1 must be divisible by 11!
            }
            int[] k2_weights = new int[] { 5, 4, 3, 2, 7, 6, 5, 4, 3, 2, 1 };
            foreach (var item in nin.Select((digit, index) =< (digit, index)))
            {
                k2 += int.Parse(item.digit.ToString()) * k2_weights[item.index];
            }
            if (k2 % 11 != 0)
            {
                return false;
            }

            return true; //k1 and k2 is now known to be both divisible with 11
        }


 
The Readme of the Nuget is added below:
 
 
 NinUtilsNorway
Summary
Util methods for Nin (National identifier number) in Norway

Note : Nin standards will be replaced by PID standard in 2032. Nin will be kept, but new Nins handed out will follow PID standard.

The following types of Nin Numbers exists, basically five types of which ordinary Nin and D-number are the most typical. They all consist of 11 digits of which the last two are control digits. (usually Modulo 11 algorithm is used):

Ordinary Nin (fødselsnummer)
D-number (temporary given to foreign workers, may span multiple years)
Help numbers H-numbers (tourists, infants, unconcious people, unidentified people, etc)
FH help numbers FH-numbers (similar to H-numbers)
DUF-number (given to asylum seekers by UDI)
About change of Nin into PID standard as of 2032 : https://www.skatteetaten.no/en/deling/opplysninger/folkeregisteropplysninger/pid/ Sample test persons can be retrieved from here which was helpful in building the util methods. https://skatteetaten.github.io/folkeregisteret-api-dokumentasjon/test-for-konsumenter

See useful list of definitions here: https://www.ehelse.no/standardisering/standarder/identifikatorer-for-personer And DUF-numbers (UDI) : https://www.udi.no/ord-og-begreper/duf-nummer/ Note - Gender calculation will not necessarily be possible after 2032, as you are not guaranteed that Nin will contain correct gender information when PID is introduced. People will keep their Nin as before, but the semantic of Gender where the ninth digit (last of the three digits of 'individual number' is even number means = FEMALE and odd number = MALE is halted after 2032. Newborns and new Nin (PID) will be gender-less, i.e. you cannot read gender out of Nin handed out after 2032.

NinUtilsNorway.NinUtilsNorway.GetGender(System.String)"
        <summary>
        Resolves gender from nin. Rule is that the first six digits are the birth date
        DDMMYYYY followed by 3 'individual digits' (individnummer) and finally two
        control digits (kontrollsiffer). The third digit of 'individual digits' are the 
        indicator for gender. If even number, female individual, if odd number, male individual.
        </summary>
        <param name="nin"></param>
        <returns></returns>
        <remarks>Documentation about Norwegian Nin structure is here<see href="https://www.skatteetaten.no/person/folkeregister/fodsel-og-navnevalg/barn-fodt-i-norge/fodselsnummer/"/></remarks>
    </member>
NinUtilsNorway.NinUtilsNorway.IsDufNumber(System.String,NinUtilsNorway.IDateTimeNowProvider)
        <summary>
        Checks if this is a DUF-number. These numbers are given by UDI 
        The check is only checking it is a number with 12 digits. The number must also 
        reside in UDI data systems, which this method do not check.
        </summary>
        <returns></returns>
        <remarks>Se notes from eHelse here: <see href="https://www.ehelse.no/standardisering/standarder/identifikatorer-for-personer#DUF-nummer"/></remarks>
    </member>
NinUtilsNorway.NinUtilsNorway.IsHelpNumber(System.String,System.Boolean)
        <summary>
        Checks if this is a help number H-number. The default convention for H-number is that it we add 
        the number 4 to the third digit 
        </summary>
        <param name="useEightNineConvention">Use special convention that if the first digit is 8 or 9, it signals 
        a Help Number. Note - this usually designates a FH-help number instead</param>
        <returns></returns>
    </member>
NinUtilsNorway.NinUtilsNorway.IsFHNumber(System.String)
        <summary>
        FH numbers are developed by KITH as a proposal established as a standard 18.01.2010. It is similar to Nin 
        fødselsnumre with 11 digits, and the first digit is 8 or 9. The numbers in position 2 - 9 are generated
        as random numbers. This standard conceals also gender, birthdate or which order the number is provided.
        The algorithms allows about 200 million numbers minus 17% of these due to incorrect control digits (last two digits). 
        Examples of people getting a FH-number are tourists, newborn (infants), unconcious people not identified, 
        unidentified people or similar reasons that a fødselsnummer Nin or D-Number is not available. 
        </summary>
        <param name="number"></param>
        <returns></returns>
    </member>
M:NinUtilsNorway.NinUtilsNorway.IsDNumber(System.String)
        <summary>
        Returns true if a person is having a D-number. A d-number is given to foreign workers in 
        Norway as a temporary identifier during their work period. It is similar to a ordinary Nin (fødselsnummer), but 
        for the first digit in the nin, we add 4. This gives 4,5,6,7 as possible digits for the first digits.
        A lot of other characteristics of D-number are similar to ordinary Nin, including the two control digits follow same rules.
        </summary>
        <param name="nin"></param>
        <returns></returns>
    </member>
NinUtilsNorway.NinUtilsNorway.GetAge(System.String,NinUtilsNorway.IDateTimeNowProvider)
        <summary>
        Calculates age from Nin
        </summary>
        <param name="nin"></param>
        <param name="nowTimeProvider">Provide an implementation to override now time. 
        Useful for mocking</param>
        <returns></returns>
        <remarks>About individual numbers - the 7-9 digits of Nin - and rules of centuries. 
        See explanation here: <see href="https://no.wikipedia.org/wiki/F%C3%B8dselsnummer" /></remarks>
NinUtilsNorway.NinutilsNorway.GetControlDigitsForNin(string nin)
     <summary>
    /// Nin are composed of two control digits at the end. We can calculate these digits. 
    /// Usage: pass in the first NINE digits of the Nin. The last two digits will then be calculated. 
    /// For given first nine digits of we calculate the control digits, last two digits of the nin
    //  Pass in the first nine digits. 11 - (the weighted sum modulo 11) is then returned for first control digit
    //  k1. And the second control digit 2 is similarly calculated, but include the first control digit also as a 
    //  self correcting mechanism.
    /// </summary>
    /// <param name="nin"></param>
    /// <returns></returns>
    </summary>
NinUtilsNorway.NinUtilsNorway.IsValidNin(string nin)
    /// <summary>
    /// Calculates validity of Nin according to modulo 11 algorithm. 
    /// </summary>
    /// <param name="nin"></param>
    /// <returns></returns>
    /// <remarks><see href="http://www.fnrinfo.no/Teknisk/KontrollsifferSjekk.aspx"
    /// Example of a Modulo-11 algorithm mathematical basis is shown here: 
    /// <see href="http://www.pgrocer.net/Cis51/mod11.html"/>
    /// </remarks>
    /// </summary>
Finally, note about testing. See :

https://skatteetaten.github.io/folkeregisteret-api-dokumentasjon/test-for-konsumenter/

For test data.

Also note that you can implement IDateTimeNowProvider to statically set "today date" for predicatable results while testing.
 
 

Thursday, 16 December 2021

AngularJs directive for clearing a text field

I wrote an AngularJs directive at work today for clearing a text field. We still use this in multiple projects for front-end (although I have worked more with Angular than AngularJs last 2-3 years). The directive ended up like this (we use Bootstrap v2.3.2) :

angular.module('formModule').directive('addClearTextFieldBtn', function ($compile) {
        function link(scope, element, attrs) {
            var targetId = attrs.id;
            var targetNgModel = attrs.ngModel;
            var minimumChars = attrs.clearTextFieldBtnMintextlength ? attrs.clearTextFieldBtnMintextlength : "1";
            var emptyValue = "''";
            var templateAppend = '<i id="clear' + targetId + '" ng-if="' + targetNgModel + ' && + ' + targetNgModel + '.length >= ' + minimumChars + '"' + 'ng-click="' + targetNgModel;
            templateAppend += ' = ' + emptyValue + '" class="glyphicon icon-remove form-control-feedback" title="Tøm innhold" style="cursor:pointer; pointer-events: all;" tooltip="clear"></i >';
            var clearButton = angular.element(templateAppend);
            clearButton.insertAfter(element);
            $compile(clearButton)(scope);
        }
        return {
            restrict: 'A',
            replace: false,
            link: link
        };
    });

Example usage inside a HTML helper in MVC for example:
add_clear_text_field_btn = "model.Icd10", data_clear_text_field_btn_mintextlength="3"
We pass in a HTML5 data value attribute to specify the minimum length to show the button to clear the field.

Sunday, 12 December 2021

Displaying errors in Event Log with Out-GridView in Powershell

A user friendly way to view errors in Event log source from Powershell.
$rawUI = $Host.UI.RawUI
$oldSize = $rawUI.BufferSize
$typeName = $oldSize.GetType( ).FullName
$newSize = New-Object $typeName (500, $oldSize.Height)
$rawUI.BufferSize = $newSize

get-eventlog -logname someacmecompanyname | where-object { $_.source -like '*someeventlogsourcename*' -and $_.EntryType -in ('Error', 'Warning', 'Critical') } | out-gridview

Wednesday, 24 November 2021

Scanning solutions for NUnit test adapter via Powershell

Checking that we have added test adapter for NUnit so that our tests in Azure Devops are run

A challenge with running tests inside Powershell can be if NUnit test adapter Nuget package is missing from the solution. If you run test using NUnit 2.x, you require NUnitTestAdapter. If you use NUnit 3.x, NUnit3TestAdapter is required. The following Powershell script can be used to check if we have added a Nuget package reference at least to one such test project in the solution. We have here some tests that will list up all PackageReference in csproj files of the solution. Note: this requires the following setup of your Nuget package references listed in the solution.
  • You have to have csproj projects in the solution
  • You must use PackageReference, i.e. list up nuget packages in the csproj file. This will not work if you instead use packages lock json format or packages.config.
The Powershell functions are these:
 
 
 
 Function Get-ProjectInSolution {
    [CmdletBinding()] param (
        [Parameter()][string]$Solution
    )
    $SolutionPath = $Solution
    $SolutionFile = Get-Item $SolutionPath
    $SolutionFolder = $SolutionFile.Directory.FullName

    Get-Content $Solution |
        Select-String 'Project\(' |
        ForEach-Object {
            $projectParts = $_ -Split '[,=]' | ForEach-Object { $_.Trim('[ "{}]') }
            [PSCustomObject]@{
                File = $projectParts[2]
                Guid = $projectParts[3]
                Name = $projectParts[1]
            }
        } |
        Where-Object File -match "csproj$" |
        ForEach-Object {
            Add-Member -InputObject $_ -NotePropertyName FullName -NotePropertyValue (Join-Path $SolutionFolder $_.File) -PassThru
        }
}

Function Get-TestProjectInSolution {
[CmdletBinding()] param (
[Parameter()][string]$Solution)
  
  $projects = & Get-ProjectInSolution $Solution
  $testProjects = $projects | Where-Object { $_.Name -like '*Test*' }
  return $testProjects
}


Function Get-PackagesInProject {
[CmdletBinding()] param (
[Parameter()][string]$ProjectFile)

Get-Content $ProjectFile | Write-Host 
}


# Get-ProjectInSolution "C:\dev\somesolution\someacme.sln" 

Function List-PackagesOfTestProjectInSolution {
[CmdletBinding()] param (
[Parameter()][string]$SolutionFile)

  & Get-TestProjectInSolution $SolutionFile | ForEach-Object {
   $filePath = $_.FullName 
   Write-Host $filePath
  (Get-Content $_.FullName | Find "<PackageReference Include")
}
}

 
    Function Get-PackagesOfTestProjectInSolution {
[CmdletBinding()] param (
[Parameter()][string]$SolutionFile)

$dict = @{}

  & Get-TestProjectInSolution $SolutionFile | ForEach-Object {
    $filePath = $_.FullName 
    # Write-Host $filePath
    if (-not $dict.ContainsKey($filePath)) {
        $dict[$filePath] = (Get-Content $_.FullName | Find "<PackageReference Include")
    }
    return $dict 
  }

}

Function Has-NunitTestAdapterPackageInTestProjectinSolution {
[CmdletBinding()] param (
[Parameter()][string]$SolutionFile) 
 $packagesDict = Get-PackagesOfTestProjectInSolution $SolutionFile
$allPackagesString = $packagesDict.Values
$isNunitTestAdapterFound = ($allPackagesString -like "*NUnit*TestAdapter*").Length -gt 0
return $isNunitTestAdapterFound
}


Get-PackagesOfTestProjectInSolution "C:\dev\someacme\someacme.sln" 

$isNunitTestAdapterPresent = Has-NunitTestAdapterPackageInTestProjectinSolution "C:\dev\someacme\somecme.sln" 

Write-Host "Is NUnit test adapter added?" $isNunitTestAdapterPresent

    
    
    
    
For example, we could run the function call : List-PackagesOfTestProjectInSolution "C:\dev\someacme\someacme.sln" And we get our lists of package references in that solution (here we only look inside projects with a name containing "Test":

Friday, 9 July 2021

Immutable lists in C# - Adding a wrapper class

This article will discuss the immutable collections in C#, more precisely immutable lists of generic type T wrapped inside a class. This makes it possible to easier use immutable lists and these lists can only be altered via functional calls. Remember that an immutable list always returns a new immutable list. For easier use, we can have a wrapper for this. First of, inside Linqpad 5, being used in this article, hit F4. In case you want to use Visual Studio instead, the same code should work there (except Linqpad's Dump method). In the tab Additional referencesNow choose Add Nuget.. Then seach for System.Collections.Immutable. After selecting this Nuget package, choose the tab Additional Namesapce Imports. Now paste this demo code:
 
 void Main()
{
	var numbersInImmutableList = new ImmutableWrappedList<int>();
	numbersInImmutableList.AddRange(new[] { 3, 1, 4, 1, 5, 9, 2 }); 
	numbersInImmutableList.AddRange(new[]{ 2, 7, 1, 8, 2, 1, 8 });
	numbersInImmutableList.RemoveAt(2);
	numbersInImmutableList.Contents.Dump(); 	
}

public class ImmutableWrappedList<T>  {
	public ImmutableList<T> _internalList;
    	public ImmutableList<T> Contents => _internalList;
	
	public ImmutableWrappedList()
	{
		_internalList = ImmutableList.Create<T>(); 	
	}
	
	public void Clear() => _internalList.Clear(); 	
	public void AddRange(IEnumerable<T> itemsToAdd) => _internalList = _internalList.AddRange(itemsToAdd);
	public void Add(T itemToAdd) => _internalList = _internalList.Add(itemToAdd);
	public void Remove(T itemToAdd) => _internalList = _internalList.Remove(itemToAdd);
	public void RemoveAt(int index) => _internalList = _internalList.RemoveAt(index);
	public void Insert(T itemToAdd, int position) => _internalList = _internalList.Insert(position, itemToAdd);
}


 
As we can see, the wrapper class can add items to the immutable collections and we also reassign the result modifying operation to the same _internalList field, which has a private setter and is initialized to an empty array in the constructor. This gives you mutability to the immutable collection without having to remember to reassign the variable, which is error prone in itself. Note - we have called the _internalList and you see that we can get thi What is the benefit of this ? Well, although we can reach into the internal collection with the Contents method here, the immutable list is still immutable. If you want to change it, you have to call specific methods here on it offered in the wrapping class. So, data-integrity wise, we have data that only can change via the methods offered in the wrapping class. A collection which is not immutable can be changed in many ways only by giving access to it. We still have control over the data via the wrapper and we make it easier to consume the immutable class by reassigning the collection.

Wednesday, 7 July 2021

Dapper - Inner Joins between two tables - Helper methods

Many developers use Entity Framework (EF) today as the library of their data access library to communicate against the database. EF is a ORM, object-relational mapper and while it boasts much functionality like change tracking and mapping relationships, Dapper at the other line of ORMs is a Micro-ORM. A Micro-ORM has less functionality, but offers usually more speed and less overhead. Dapper is a great Micro-ORM, however, writing SQL manually is often error-prone or tedious. Some purists love writing the SQL manually and be sure which SQL they send off to the DB. That is much of the point of Dapper. However, lending a hand to developers in building their SQL should still be allowed. The query compilation time added to such helper methods are miniscule anyways compared to the heavy overhead of an advanced ORM like EF. Anyways, the code in this article shows some code I am working with for building inner joins between to tables. The relationship between the two tables are 1:1 in my test case and the inner join does for now not support a where predicate filter, although adding such a filter should be easy. The source code for DapperUtils of mine is available on GitHub: https://github.com/toreaurstadboss/DapperUtils
First, we make use of SqlBuilder from DapperUtils addon lib for Dapper.

using Dapper;
using System.Collections.Generic;
using System.Linq;
using System.Text.RegularExpressions;

namespace DapperUtils.ToreAurstadIT
{
    /// <summary>
    /// Original is fetched from: https://raw.githubusercontent.com/DapperLib/Dapper/main/Dapper.SqlBuilder/SqlBuilder.cs
    /// 
    /// </summary>
    public class SqlBuilder
    {
        private readonly Dictionary<string, Clauses> _data = new Dictionary<string, Clauses>();
        private int _seq;

        private class Clause
        {
            public string Sql { get; set; }
            public object Parameters { get; set; }
            public bool IsInclusive { get; set; }
        }

        private class Clauses : List<Clause>
        {
            private readonly string _joiner, _prefix, _postfix;

            public Clauses(string joiner, string prefix = "", string postfix = "")
            {
                _joiner = joiner;
                _prefix = prefix;
                _postfix = postfix;
            }

            public string ResolveClauses(DynamicParameters p)
            {
                foreach (var item in this)
                {
                    p.AddDynamicParams(item.Parameters);
                }
                return this.Any(a => a.IsInclusive)
                    ? _prefix +
                      string.Join(_joiner,
                          this.Where(a => !a.IsInclusive)
                              .Select(c => c.Sql)
                              .Union(new[]
                              {
                                  " ( " +
                                  string.Join(" OR ", this.Where(a => a.IsInclusive).Select(c => c.Sql).ToArray()) +
                                  " ) "
                              }).ToArray()) + _postfix
                    : _prefix + string.Join(_joiner, this.Select(c => c.Sql).ToArray()) + _postfix;
            }
        }

        public class Template
        {
            private readonly string _sql;
            private readonly SqlBuilder _builder;
            private readonly object _initParams;
            private int _dataSeq = -1; // Unresolved

            public Template(SqlBuilder builder, string sql, dynamic parameters)
            {
                _initParams = parameters;
                _sql = sql;
                _builder = builder;
            }

            private static readonly Regex _regex = new Regex(@"\/\*\*.+?\*\*\/", RegexOptions.Compiled | RegexOptions.Multiline);

            private void ResolveSql()
            {
                if (_dataSeq != _builder._seq)
                {
                    var p = new DynamicParameters(_initParams);

                    rawSql = _sql;

                    foreach (var pair in _builder._data)
                    {
                        rawSql = rawSql.Replace("/**" + pair.Key + "**/", pair.Value.ResolveClauses(p));
                    }
                    parameters = p;

                    // replace all that is left with empty
                    rawSql = _regex.Replace(rawSql, "");

                    _dataSeq = _builder._seq;
                }
            }

            private string rawSql;
            private object parameters;

            public string RawSql
            {
                get { ResolveSql(); return rawSql; }
            }

            public object Parameters
            {
                get { ResolveSql(); return parameters; }
            }
        }

        public Template AddTemplate(string sql, dynamic parameters = null) =>
            new Template(this, sql, parameters);

        protected SqlBuilder AddClause(string name, string sql, object parameters, string joiner, string prefix = "", string postfix = "", bool isInclusive = false)
        {
            if (!_data.TryGetValue(name, out Clauses clauses))
            {
                clauses = new Clauses(joiner, prefix, postfix);
                _data[name] = clauses;
            }
            clauses.Add(new Clause { Sql = sql, Parameters = parameters, IsInclusive = isInclusive });
            _seq++;
            return this;
        }

        public SqlBuilder Intersect(string sql, dynamic parameters = null) =>
            AddClause("intersect", sql, parameters, "\nINTERSECT\n ", "\n ", "\n", false);

        public SqlBuilder InnerJoin(string sql, dynamic parameters = null) =>
            AddClause("innerjoin", sql, parameters, "\nINNER JOIN ", "\nINNER JOIN ", "\n", false);

        public SqlBuilder LeftJoin(string sql, dynamic parameters = null) =>
            AddClause("leftjoin", sql, parameters, "\nLEFT JOIN ", "\nLEFT JOIN ", "\n", false);

        public SqlBuilder RightJoin(string sql, dynamic parameters = null) =>
            AddClause("rightjoin", sql, parameters, "\nRIGHT JOIN ", "\nRIGHT JOIN ", "\n", false);

        public SqlBuilder Where(string sql, dynamic parameters = null) =>
            AddClause("where", sql, parameters, " AND ", "WHERE ", "\n", false);

        public SqlBuilder OrWhere(string sql, dynamic parameters = null) =>
            AddClause("where", sql, parameters, " OR ", "WHERE ", "\n", true);

        public SqlBuilder OrderBy(string sql, dynamic parameters = null) =>
            AddClause("orderby", sql, parameters, " , ", "ORDER BY ", "\n", false);

        public SqlBuilder Select(string sql, dynamic parameters = null) =>
            AddClause("select", sql, parameters, " , ", "", "\n", false);

        public SqlBuilder AddParameters(dynamic parameters) =>
            AddClause("--parameters", "", parameters, "", "", "", false);

        public SqlBuilder Join(string sql, dynamic parameters = null) =>
            AddClause("join", sql, parameters, "\nJOIN ", "\nJOIN ", "\n", false);

        public SqlBuilder GroupBy(string sql, dynamic parameters = null) =>
            AddClause("groupby", sql, parameters, " , ", "\nGROUP BY ", "\n", false);

        public SqlBuilder Having(string sql, dynamic parameters = null) =>
            AddClause("having", sql, parameters, "\nAND ", "HAVING ", "\n", false);

        public SqlBuilder Set(string sql, dynamic parameters = null) =>
             AddClause("set", sql, parameters, " , ", "SET ", "\n", false);

    }
}

Using SqlBuilder, we can define a Sql template and add extension methods and helper methods required to build and retrieve the inner join. The helper methods in use are added also below the extension method InnerJoin. Make note that we use SqlBuilder here to do much of the SQL template processing to end up
with the SQL that is sent to the DB (RawSql property of SqlBuilder instance).

        /// <summary>
        /// Inner joins the left and right tables by specified left and right key expression lambdas.
        /// This uses a template builder and a shortcut to join two tables without having to specify any SQL manually
        /// and gives you the entire inner join result set. It is an implicit requirement that the <paramref name="leftKey"/>
        /// and <paramref name="rightKey"/> are compatible data types as they are used for the join.
        /// This method do for now not allow specifying any filtering (where-clause) or logic around the joining besides
        /// just specifying the two columns to join.
        /// </summary>
        /// <typeparam name="TLeftTable">Type of left table</typeparam>
        /// <typeparam name="TRightTable">Type of right table</typeparam>
        /// <param name="connection">IDbConnection to the DB</param>
        /// <param name="leftKey">Member expression of the left table in the join</param>
        /// <param name="rightKey">Member expression to the right table in the join</param>
        /// <returns>IEnumerable of ExpandoObject. Tip: Iterate through the IEnumerable and save each ExpandoObject into a variable of type dynamic to access the variables more conveniently if desired.</returns>
        public static IEnumerable<ExpandoObject> InnerJoin<TLeftTable, TRightTable>(this IDbConnection connection, 
            Expression<Func<TLeftTable, object>> leftKey, Expression<Func<TRightTable, object>> rightKey)
        {
            var builder = new SqlBuilder();
            string leftTableSelectClause = string.Join(",", GetPublicPropertyNames<TLeftTable>("l"));
            string rightTableSelectClause = string.Join(",", GetPublicPropertyNames<TRightTable>("r"));
            string leftKeyName = GetMemberName(leftKey);
            string rightKeyName = GetMemberName(rightKey); 
            string leftTableName = GetDbTableName<TLeftTable>();
            string rightTableName = GetDbTableName<TRightTable>(); 
            string joinSelectClause = $"select {leftTableSelectClause}, {rightTableSelectClause} from {leftTableName} l /**innerjoin**/";
            var selector = builder.AddTemplate(joinSelectClause);
            builder.InnerJoin($"{rightTableName} r on l.{leftKeyName} = r.{rightKeyName}");
            var joinedResults = connection.Query(selector.RawSql, selector.Parameters)
                .Select(x => (ExpandoObject)DapperUtilsExtensions.ToExpandoObject(x)).ToList();
            return joinedResults;
        }
        
          private static string[] GetPublicPropertyNames<T>(string tableQualifierPrefix = null) {
            return typeof(T).GetProperties(System.Reflection.BindingFlags.Public | System.Reflection.BindingFlags.Instance)
                 .Where(x => !IsNotMapped(x))
                 .Select(x => !string.IsNullOrEmpty(tableQualifierPrefix) ? tableQualifierPrefix + "." + x.Name : x.Name).ToArray();
        }

     private static bool IsNotMapped(PropertyInfo x)
        {
            var notmappedAttr = x.GetCustomAttributes<NotMappedAttribute>()?.OfType<NotMappedAttribute>().FirstOrDefault();
            return notmappedAttr != null;
        }
       /// <summary>
        /// Returns database table name, either via the System.ComponentModel.DataAnnotations.Schema.Table attribute
        /// if it exists, or just the name of the <typeparamref name="TClass"/> type parameter. 
        /// </summary>
        /// <typeparam name="TClass"></typeparam>
        /// <returns></returns>
        private static string GetDbTableName<TClass>()
        {
            var tableAttribute = typeof(TClass).GetCustomAttributes(typeof(TableAttribute), false)?.FirstOrDefault() as TableAttribute;
            if (tableAttribute != null)
            {
                if (!string.IsNullOrEmpty(tableAttribute.Schema))
                {
                    return $"[{tableAttribute.Schema}].[{tableAttribute.Name}]";
                }
                return tableAttribute.Name;
            }
            return typeof(TClass).Name;
        }     

        private static string GetMemberName<T>(Expression<Func<T, object>> expression)
        {
            switch (expression.Body)
            {
                case MemberExpression m:
                    return m.Member.Name;
                case UnaryExpression u when u.Operand is MemberExpression m:
                    return m.Member.Name;
                default:
                    throw new NotImplementedException(expression.GetType().ToString());
            }
        }

        /// <summary>
        /// Returns database table name, either via the System.ComponentModel.DataAnnotations.Schema.Table attribute
        /// if it exists, or just the name of the <typeparamref name="TClass"/> type parameter. 
        /// </summary>
        /// <typeparam name="TClass"></typeparam>
        /// <returns></returns>
        private static string GetDbTableName<TClass>()
        {
            var tableAttribute = typeof(TClass).GetCustomAttributes(typeof(TableAttribute), false)?.FirstOrDefault() as TableAttribute;
            if (tableAttribute != null)
            {
                if (!string.IsNullOrEmpty(tableAttribute.Schema))
                {
                    return $"[{tableAttribute.Schema}].[{tableAttribute.Name}]";
                }
                return tableAttribute.Name;
            }
            return typeof(TClass).Name;
        }     

        public static ExpandoObject ToExpandoObject(object value)
        {
            IDictionary<string, object> dapperRowProperties = value as IDictionary<string, object>;
            IDictionary<string, object> expando = new ExpandoObject();
            if (dapperRowProperties == null)
            {
                return expando as ExpandoObject;
            }
            foreach (KeyValuePair<string, object> property in dapperRowProperties)
            {
                if (!expando.ContainsKey(property.Key))
                {
                    expando.Add(property.Key, property.Value);
                }
                else
                {
                    //prefix the colliding key with a random guid suffixed 
                    expando.Add(property.Key + Guid.NewGuid().ToString("N"), property.Value);
                } 
            }
            return expando as ExpandoObject;
        }       
        
        

Here are some Nuget packages in use in the small lib functions here are in test project too:

	   <!-- lib project .NET 5 -->
       <PackageReference Include="Dapper" Version="2.0.90" />
	   <PackageReference Include="Microsoft.CSharp" Version="4.7.0" />
	   <PackageReference Include="System.ComponentModel.Annotations" Version="5.0.0" />
       
       <!-- test project-->
        <PackageReference Include="FluentAssertions" Version="5.10.3" />
		<PackageReference Include="Microsoft.CSharp" Version="4.7.0" />
		<PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="3.1.16" />
		<PackageReference Include="Microsoft.Extensions.Options.ConfigurationExtensions" Version="3.1.16" />
		<PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.10.0" />
		<PackageReference Include="Newtonsoft.Json" Version="13.0.1" />
		<PackageReference Include="NUnit" Version="3.13.2" />
		<PackageReference Include="NUnit3TestAdapter" Version="4.0.0" />
		<PackageReference Include="System.ComponentModel.Annotations" Version="5.0.0" />
		<PackageReference Include="System.Data.SqlClient" Version="4.8.2" />

Two unit tests shows how easier syntax we get with this helper method. The downside is that you cant fully control the sql yourself, but the benefit is quicker to implement.
  
       [Test]
        public void InnerJoinWithManualSqlReturnsExpected()
        {
            var builder = new SqlBuilder();
            var selector = builder.AddTemplate("select p.ProductID, p.ProductName, p.CategoryID, c.CategoryName, s.SupplierID, s.City from products p /**innerjoin**/");
            builder.InnerJoin("categories c on c.CategoryID = p.CategoryID");
            builder.InnerJoin("suppliers s on p.SupplierID = s.SupplierID");
            dynamic joinedproductsandcategoryandsuppliers = Connection.Query(selector.RawSql, selector.Parameters).Select(x => (ExpandoObject)DapperUtilsExtensions.ToExpandoObject(x)).ToList();
            var firstRow = joinedproductsandcategoryandsuppliers[0];
            Assert.AreEqual(firstRow.ProductID + firstRow.ProductName + firstRow.CategoryID + firstRow.CategoryName + firstRow.SupplierID + firstRow.City, "1Chai1Beverages1London");
        }

        [Test]
        public void InnerJoinWithoutManualSqlReturnsExpected()
        {
            var joinedproductsandcategory = Connection.InnerJoin<Product, Category>(l => l.CategoryID, r => r.CategoryID);
            dynamic firstRow = joinedproductsandcategory.ElementAt(0);
            Assert.AreEqual(firstRow.ProductID + firstRow.ProductName + firstRow.CategoryID + firstRow.CategoryName + firstRow.SupplierID, "1Chai1Beverages1");
        }
  
Our POCO classes used in the tests are these two. We use the Nuget package System.ComponentModel.Annotations and attributes TableName and NotMapped to control the SQL built here to specify the DB table name for the POCO (if they are the same, the name of the type is used as fallback if attribute TableName is missing) and NotMapped in case there are properties like relationship properties ("navigation properties in EF for Dapper") that should not be used in the SQL select clause.
 
 using System.ComponentModel.DataAnnotations.Schema;

namespace DapperUtils.ToreAurstadIT.Tests
{
    [Table("Products")]
    public class Product
    {
        public int ProductID { get; set; }
        public string ProductName { get; set; }
        public int? SupplierID { get; set; }
        public int? CategoryID { get; set; }
        public string QuantityPerUnit { get; set; }
        public decimal? UnitPrice { get; set; }
        public short? UnitsInStock { get; set; }
        public short? UnitsOnOrder { get; set; }
        public short? ReorderLevel { get; set; }
        public bool? Discontinued { get; set; }
        [NotMapped]
        public Category Category { get; set; }
    }
}

using System.ComponentModel.DataAnnotations.Schema;

namespace DapperUtils.ToreAurstadIT.Tests
{
    [Table("Categories")]
    public class Category
    {
        public int CategoryID { get; set; }
        public string CategoryName { get; set; }
        public string Description { get; set; }
        public byte Picture { get; set; }
    }
}

 
In the end, we have a easy way to do a standard join. An improvement here could be the following:
  • Support for where predicates to filter the joins
  • More control on the join condition if desired
  • Support for joins accross three tables (or more?) - SqlBuilder already supports this, what is missing is lambda expression support for Intellisense support
  • What if a property does not match against db column ? Should support ColumnName attribute from System.ComponentModel.DataAnnotations.
  • Investigate other join types such as left outer joins - this should be just a minor adjustment actually.