Title Configuration

Title administrators can customize the crash collection in two ways: a JSON configuration file intended for the majority of metadata/bug filing fields as well as a C# script file for additional advanced customizations. The JSON file can be most easily edited in a developer tool like Visual Studio Code or Visual Studio since we make use of a JSON schema to provide auto-complete support and error validation in the editor.

Initial setup

First, use the BugRepository section to specify your Azure DevOps organization and desired work item types.

"BugRepository": {
    "Repository": {
        "AdoRepository": {
            "BaseUri": "https://dev.azure.com/organization",
            "Project": "ProjectName"
        }
    },
    "BugInstanceItemType": "Bug", // this is the default
    "CrashInstanceItemType": "Crash Instance" // this is the default
}

If the target should be a Jira instance, instead of "AdoRepository" it can be specified like so:

"JiraRepository": {
    "BaseUri": "https://organization.atlassian.net/",
    "Project": "PROJECTKEY"
}

Next, fill in the Metadata section with extracted or hard-coded field values.

"Metadata": [
    {
        "Name": "BucketId",
        "Query": "Failure.BucketId",
        "QueryType": "WatsonAnalysis"
    },
    {
        "Name": "FailFast",
        "Source": "WatsonAnalysis.XML",
        "Query": "ANALYSIS.KVS.KV[?\"@K\"=='FailFast.Name'].\"@V\"|[0]",
        "QueryType": "JMESPath",
        "AppliesTo": "bugs"
    },
    {
        "Name": "System.AssignedTo",
        "Source": "developer@example.com",
        "QueryType": "StaticValue",
        "AppliesTo": "both"
    },
]

In the above example, the parsed WatsonAnalysis object is used to get the failure ID, a path query is used to extract a custom piece of data, and finally a hard-coded value is set for the assignee.

To specify field values for multi-choice Jira fields such as priority, status, or assignee, there are two options: using "field.id" as the metadata name if you know the value's corresponding ID in Jira (e.g. 1), or else we'll pass the given value to Jira to decode from its name (e.g. High). Custom fields must also be specified by ID, such as "customfield_10000". A comment can be added to help explain:

{
    "Name": "customfield_10000.id",
    "Source": "3",
    "QueryType": "StaticValue",
    "Comment": "Set severity to 3 for Medium."
},

The crash ID will also be stored in a field (by default, "Custom.CrashID" in Azure DevOps) for CrashCABN dupe checks, though this is not required if only using pull ingestion with default MaxCrashesPerFailure of 1. This field may be changed in the BugRepository section. Jira doesn't have a crash ID field configured by default, so you'll only be able to use pull ingestion with defaults. Note the generated Jira query uses = to find existing crashes by ID, but this can be changed to ~ by setting IsCloud to true in the JiraRepository section.

By default the full WatsonAnalysis.xml file contents will be attached to each crash instance, and you can also add a hyperlink relation to the original Watson source URL like so:

"BugRepository": {
    "Repository": {
        "AdoRepository": {
            "BaseUri": "https://dev.azure.com/organization",
            "Project": "ProjectName",
            "AddWatsonAnalysisAsHyperlink": true
        }
    }
}

The C# script file will be managed by CrashCABN engineers for now until we have a secure process to allow end-user customization.

Onboarding

  1. In addition to the JSON config and C# script file, titles need to have a secret added to the CrashCABNv2 KeyVault for the bug filer access token.
    • The secret name can be either {TitleName}-BugFilerPAT or Azure DevOps can also use {OrgName}-BugFilerPAT ('-' characters must be removed from the name).
    • With Jira, the format should be "{username}:{token}", while Azure DevOps should only have the token.
    • If the crash storage account is not "{TitleName}crashstorage", specify it in the secret {TitleName}-StorageAccountName.
    • Further details about CrashCABN's storage account usage can be found here. Note an expiry policy for non-retail crashes of 3 months may also be used to limit storage costs.
  2. After adding or updating secrets in key vault, restart the crashcabntitleingestion function app.
  3. Create and upload the JSON configuration file.
    • Specify the bug repository URL, project, and desired work item types.
    • Specify any required fields (such as Area Path) or additional data to extract.
    • Download the title configuration tool if needed.
    • Run this command, substituting "MyConfig" and "MyTitle" for the appropriate values: 
      # PowerShell
Get-Content MyConfig.json | .\TitleConfigurationClient.exe set --title-name MyTitle

# bash
.\TitleConfigurationClient.exe set --title-name MyTitle < MyConfig.json

# Command Prompt: Unsupported
  4. Create and upload the C# script file.
    • Copy the default WatsonToAzureDevOps.cs from the test/TestFiles/ folder.
    • Run this command, substituting "MyMetadata" and "MyTitle" for the appropriate values: 
      # PowerShell
Get-Content MyMetadata.cs | .\TitleConfigurationClient.exe metadata --set --title-name MyTitle

# bash
.\TitleConfigurationClient.exe metadata --set --title-name MyTitle < MyMetadata.cs

# Command Prompt: Unsupported
  5. Run a test of pull ingestion, with .\TitleConfigurationClient.exe pull-ingest -t --title-name MyTitle
  6. Monitor the bug repository for new work items, or check the crashcabntitleingestionappinsights for traces, events, or exceptions.
    • You may also try downloading a crash to validate the web portal and API can find the title's V2 crashes.
  7. Once the configuration is working, enable pull ingestion with .\TitleConfigurationClient.exe pull-ingest -e --title-name MyTitle

Pull Ingestion

To enable pull ingestion, simply add a configuration section to your JSON file like so:

"PullIngest": {
    "Apps": [
        {
            // the app ID or executable and build versions to query
            "AppName": "microsoft.titlename",
            "Versions": [
                "1.2.3.0",
                "1.11.22222.0"
            ],
            // optional prefix for filed bugs/crashes
            "BuildType": "Retail",
            // optional overrides of global defaults
            "Settings": {
                "QueryRange": {
                    "Days": 28,
                    "Hours": 0
                }
            }
        }
    ],
    "Cadence": {
        "Days": 0,
        "Hours": 6
    },
    "Defaults": {
        "QueryRange": {
            "Days": 14,
            "Hours": 0
        },
        "MaxCrashesToPull": 5, // limit how many top failures/bugs to pull per app version
        "MaxCrashesPerFailure": 1, // limit how many crashes to pull per failure/app version
        "MinApportionedHits": 5, // limit the minimum number of failure hits
        "IncludeFailures": [ "*mygame.dll*" ] // only include failures matching these patterns
        "ExcludeFailures": [ "*radar_leak*" ] // ignore any failures matching these patterns
    }
}

Builds to Query

One method to configure the builds is to use the tools to query Watson for the title's app IDs and versions to hard-code in the JSON. This ensures crash collection for each configured build, but it can be tedious to maintain.

Alternatively, the version numbers may contain '*' to indicate "all versions" or "all sub-versions" via "x.*", "x.y.*", or "x.y.z.*". In this case, a single query will be made to collect the top failures for all matching versions -- so with this approach, lower-volume builds may have their "top failures" ignored if others are crashing at high rates.

Repeated versions will be ignored, and their order is also randomized as decided by the Shuffle setting.

Note that each AppName may be specified multiple times to customize the settings per version, such as to increase the crash instance count for a specific set:

"Apps": [
    {
        "AppName": "microsoft.titlename",
        "Versions": [ "*" ],
    },
    {
        "AppName": "microsoft.titlename",
        "Versions": [ "1.19.*", "1.20.0.*" ],
        "Settings": {
            "MaxCrashesPerFailure": 5,
            "QueryRange": {
                "Days": 7,
                "Hours": 0
            }
        }
    }
],

After configuring the versions, you may enable pull ingestion for your title via the TitleConfigurationClient or MetadataTestUI tool.

Additional File Collection and Retail Detection

Use the AdditionalFiles section to specify custom files, either to make them available for parsing during metadata extraction, attachment to the crash instance, or as a development flag to mark crashes as non-retail. Crashes must also be from a development sandbox in order to be treated as non-retail.

By default, custom files will be available for parsing and attachment on development crashes. Also note that "glob" file prefixes may be specified such as "debug_log_*" or "DevInfo_*", but not suffixes like "*.txt". For example:

"AdditionalFiles": [
    {
      "FileName": "debug_log_*",
      "IsDevFlagFile": true,
      "AttachForDevCrashes": false,
      "ParseForDevCrashes": false
    },
    {
      "FileName": "DevInfo_*",
      "AttachForRetailCrashes": true,
      "ParseForRetailCrashes": true
    }
]

Custom Bucketing Field and Template

The default behavior uses the normalized failure bucket ID, such as invalid_pointer_read!GameWorld::update, to organize crashes into bug work items, where the bug title matches this ID (with a "[CrashCABN]" prefix). The "normalization" step simply removes the exception code and process name like in invalid_pointer_read_c0000005_mygame.exe!GameWorld::update. Please note while duplicate bugs should not occur regularly, there are ADO limits of 10000 revisions or 1000 children which will require new bugs to be created.

This behavior means that when users change the bug title, then CrashCABN will organize additional crashes into a new bug. If changing the title is desireable, the bucket value can instead be stored in a custom field like so:

"BugRepository": {
    ...
    "BucketByFieldName": "Custom.CrashCABNBucket"
}

In this case you'll also have to map a metadata value for the title, which can be accomplished in the JSON:

{
    "Name": "System.Title",
    "Query": "[CrashCABN] {FailureId}",
    "QueryType": "Bucket"
},

The format of the bucket may also be customized via the BucketByTemplate setting, which defaults to [CrashCABN]{BuildType} {MissingSymbols}{LegacyFailureID}, where:

  • BuildType is the optional value set in the PullIngest configuration.
  • MissingSymbols contains Missing Symbols Bug Bucket: if the failure symbol is unknown.
  • MissingSymbolsModule contains Missing Symbols Bug Bucket (module.dll): if the failure symbol is unknown.
  • FailureType is the failure bucket ID's preamble such as invalid_pointer_read.
  • FailureSymbol is the failure bucket ID's postscript, such as GameWorld::update.
  • FailureID is the normalized failure bucket ID, such as invalid_pointer_read!GameWorld::update.
  • LegacyFailureID is the normalized failure bucket ID, with special characters converted to _.

So for example to ignore the failure type and organize all crashes from GameWorld::update into the same bug, you can configure the template like so:

"BugRepository": {
    ...
    "BucketByTemplate": "[CrashCABN] {FailureSymbol}"
}

There are also multiple formatting options available to change the capitalization of the interpolated bucket data, which can be applied like {FailureType|TitleCase}:

  • LowerCase and UpperCase update the casing of all characters and also convert underscores to spaces.
  • SentenceCase changes a string like invalid_pointer_read to Invalid pointer read.
  • TitleCase changes a string like invalid_pointer_read to Invalid Pointer Read.
  • SHA256 changes a string like invalid_pointer_read to a hash 07phr0agng5xnr64cwdyqxn4l7i2mns57ih3zjzlj0mhawqwfkdv.

Finally, note that this setting does not affect the GetBucket C# script method, so in that case you can supply a template:

yield return new MetadataDefinition(string.Empty, QueryType.StaticValue, "System.Title", GetBucket(Crash, analysis, Logger, DefaultMaxTitleLength, "[CrashCABN] {FailureSymbol}"), AppliesToEnum.Both);

Bug Filters

By default, all bugs in the bug repository are considered valid and relevant to the title. This means that resolved/closed bugs may see new crash instances appended to them, and if multiple titles exist in the bug repository, their crashes may be organized in overlapping bugs.

These behaviors can however be modified using the SearchFilter and ParentBugFilter options:

"BugRepository": {
    ...
    "SearchFilter": {
        "Key": "Custom.Product",
        "Value": "Contoso"
    },
    "ParentBugFilter": {
        "Key": "System.State",
        "Value": [ "New", "Active" ]
    }
}

In this example, SearchFilter ensures that searches for existing bugs will only match ones for the correct product, which enables CrashCABN to search for and file bugs for multiple products without overlap. The product field should also be mapped in the Metadata section. Note that SearchFilter only supports a single value.

The ParentBugFilter then further limits the matched bugs to only those in the new or active states. CrashCABN will still count any crash instances filed in closed/resolved bugs, but it will not append new ones to them.

In cases where CrashCABN finds an existing bug but cannot append new crashes, it will file a new bug with a comment including links to any older instances.

This page was last modified on July 03 2024, 07:45 PM (UTC).