Technologies

This document summarizes the various tools used to help with CrashCABN development.

F# parser

For the V2 rewrite of the crash analyzer and WatsonAnalysis.xml parser, we first considered a dynamic type/method approach which allowed for a convenient way for C# code to get various parts and sections of the XML. We also considered using XPath directly or XML=>JSON conversion followed by JsonPath or JmesPath.

Instead we decided a small amount of F# powered by the FSharp.Data library would be preferred as a type-safe and easy-to-write codebase based on an auto-generated XML parser from its schema.

The C# code that interacts with it uses the Pather library to fetch data out of the parsed Analysis objects from a query like "Failure.BucketId". A simpler test example of the F# WatsonParser can be found here.

OData client

For the pull ingestion pipeline, where crashes come from a periodic query of the data source instead of a push process where it sends us an event for each crash to ingest, we needed a mechanism to call Watson's OData API from a C# tool, and the built-in "connected service" feature provided a very convenient to use auto-generated client based on the OData API's metadata XML. A quick guide on how this works can be found here.

Here is an example of the auto-generated C# code and the metadata XML. This can be easily used from C# like so.

One gotcha to be aware of is, when debugging, the "results" enumeration can not be listed multiple times. However you can simply inspect the "query" in the debugger to see the URL and then open it in a web browser if you need to see the original XML.

OpenAPI/Swagger

For title ingestion V2 and the pull ingestion pipeline, we needed to build an end-user accessible configuration tool to manage their title's settings and pull ingestion options. In order to deliver a working solution as quickly as possible, we decided against traditional MVC/HTML web UI and instead worked towards a re-usable web API and C# client.

One popular technology for web APIs is OpenAPI/Swagger, and ASP.NET can auto-generate the supporting metadata for it from your controllers. Further, it will inspect your API's request/response objects and include JSON metadata for them too, so your clients will be able to auto-generate type-safe code to interact with the API. More info on this can be found in the ASP.NET docs here.

In CrashCABN a few customizations had to be applied to change it to use JSON.NET for our types instead of System.Text.Json and to use PascalCase instead of the default camelCase for JSON properties, but once those changes were applied the client just worked.

That said, some care may need to be taken for more complex APIs that return IActionResult for example, you'll need to add attributes to your controller so Swashbuckle (the server-side API generator) can know what types/response codes to expect and so NSwag can generate appropriate C# client code.

There are many options available to customize how NSwag generates code, such as the namespace or System data types used, as well as a free desktop app called NSWagStudio. We do not need to use any of these since with CrashCABN on a recent .NET/C# version, source generators and the OpenApiReference csproj item work.

One last convenient technology that comes with Swagger is the API explorer, found at https://localhost:5001/swagger/, which lets you see all your APIs and example request/responses in one place, as well as perform queries and generate URLs for them.

Console App Framework

Command line tools are a convenient way to prototype and build quick functionality without working on more expensive/time-consuming UI design, but good command line interfaces can be challenging to create too.

.NET Core provides a good foundation for this with Generic Host, which shares a similar "app startup" mechanism as ASP.NET and allows for easy integration of dependency injection/configuration as well as code re-use between tools and services.

However it does not provide much built-in support for command line arguments, and building tools with it may require significant boilerplate. As its author describes, "ConsoleAppFramework is like ASP.NET Core MVC in CLI Applications, no needs boilerplate."

The benefits for CrashCABN are two-fold:

• Significant code sharing between tools and the Azure Functions and Web projects. See for example how Program.cs required very little changes from the existing Program.cs built for Functions

• Easy to write command tool APIs with auto-parsed arguments and auto-generated help text:

    Usage: TitleConfigurationClient [options...]
  
    Options:
    -titleName <String>     (Required)
  
    Usage: TitleConfigurationClient <Command>
  
    Commands:
    list          Lists all title configurations
    set           Sets the title configuration
    pullingest    Enables (-e), disables (-d), tests (-t), or checks if pull ingestion is enabled for a title configuration
    help          Display help.
    version       Display version.
  

JSON schema

The title configuration JSON has a corresponding JSON schema, which can be used to validate the JSON and provide syntax highlighting and auto-completion in compatible editors such as Visual Studio and VS Code. It may be possible to auto-generate the schema from the C# types, but the current version is maintained by hand.

The web API uses the JSON schema and corresponding version number to validate that clients are up-to-date and they only use valid configurations.

C# script

Since V2 uses a unified deployment of the service that applies to all titles, it needed a replacement for the advanced pieces of customization that would previously have been added to the title's CrashCABN fork, and this is where it uses C# scripting via the Roslyn compiler. The standard WatsonToAzureDevOps.cs script can be found here.

In the script, a few global objects and helper methods are available, as defined in ScriptGlobals:

• Attachments, an IAttachmentContainer which may be indexed by CommonAttachmentNames.
• Crash, a CrashInstance containing details about the crash.

Strings

• Label and SpacedLabel, used to generate prettified bug descriptions.
• TruncateString, used to shorten the bug descriptions to fit limits.

Watson

• BuildCrashApiDownloadLink, used to create a crash download link
• GetApportionedHits, used to get the number of apportioned hits for the crash (if known)
• GetNormalizedBucketId, used to remove extraneous information from the failure bucket ID
• GetTitle, used to generate a bug title based off the failure bucket ID
• IsMissingSymbols, used to determine if the failure bucket is for missing symbols
• Retail and NonRetailString, used for crash download links (TBD)
• StripProcessName, used to remove the process name from the failure bucket ID

---

This page was last modified on February 03 2023, 07:24 PM (UTC).