Support Log Search Alerts

[Ryan-Palmer]

In the LogAnalytics docs, if you set a daily usage cap it is recommended to also set an alert if you hit that cap.

They provide an example log search alert configuration here.

The portal set up is more or less just this:

It would be great to deploy this alert with Farmer (plus set up alerts on any other custom queries we might need).

If I can work out what to do, I'm happy to have a go at adding it :)

[Ryan-Palmer]

Ah, looking at the ARM docs for log search alerts I just realised that the alert builder in Farmer is actually building a Microsoft.Insights/metricAlerts resource, but log search alerts use the Microsoft.Insights/scheduledQueryRules resource which I don't think exists in Farmer yet. Just picking through the docs / source etc now trying to work out what is going on.

Metric alert (just 'alert' in Farmer) :

{
      "type": "Microsoft.Insights/metricAlerts",
      "apiVersion": "2018-03-01",
      "name": "[parameters('alertName')]",
      "location": "global",
      "properties": {
        "description": "[parameters('alertDescription')]",
        "severity": "[parameters('alertSeverity')]",
        "enabled": "[parameters('isEnabled')]",
        "scopes": [
          "[parameters('resourceId')]"
        ],
        "evaluationFrequency": "[parameters('evaluationFrequency')]",
        "windowSize": "[parameters('windowSize')]",
        "criteria": {
          "odata.type": "Microsoft.Azure.Monitor.SingleResourceMultipleMetricCriteria",
          "allOf": [
            {
              "name": "1st criterion",
              "metricName": "[parameters('metricName')]",
              "dimensions": [],
              "operator": "[parameters('operator')]",
              "threshold": "[parameters('threshold')]",
              "timeAggregation": "[parameters('timeAggregation')]",
              "criterionType": "StaticThresholdCriterion"
            }
          ]
        },
        "actions": [
          {
            "actionGroupId": "[parameters('actionGroupId')]"
          }
        ]
      }
    }

Log alert (scheduledQueryRule, not in Farmer afaik):

{
      "type": "Microsoft.Insights/scheduledQueryRules",
      "apiVersion": "2021-08-01",
      "name": "[parameters('alertName')]",
      "location": "[parameters('location')]",
      "tags": {},
      "properties": {
        "description": "[parameters('alertDescription')]",
        "severity": "[parameters('alertSeverity')]",
        "enabled": "[parameters('isEnabled')]",
        "scopes": [
          "[parameters('resourceId')]"
        ],
        "evaluationFrequency": "[parameters('evaluationFrequency')]",
        "windowSize": "[parameters('windowSize')]",
        "criteria": {
          "allOf": [
            {
              "query": "[parameters('query')]",
              "metricMeasureColumn": "[parameters('metricMeasureColumn')]",
              "resourceIdColumn": "[parameters('resourceIdColumn')]",
              "dimensions": [],
              "operator": "[parameters('operator')]",
              "threshold": "[parameters('threshold')]",
              "timeAggregation": "[parameters('timeAggregation')]",
              "failingPeriods": {
                "numberOfEvaluationPeriods": "[parameters('numberOfEvaluationPeriods')]",
                "minFailingPeriodsToAlert": "[parameters('minFailingPeriodsToAlert')]"
              }
            }
          ]
        },
        "muteActionsDuration": "[parameters('muteActionsDuration')]",
        "autoMitigate": "[parameters('autoMitigate')]",
        "checkWorkspaceAlertsStorageConfigured": "[parameters('checkWorkspaceAlertsStorageConfigured')]",
        "actions": {
          "actionGroups": [
            "[parameters('actionGroupId')]"
          ],
          "customProperties": {
            "key1": "value1",
            "key2": "value2"
          }
        }
      }
    }

[Ryan-Palmer]

Did the quick hack test, got it working with the JSON builder

let scheduledQueryRule
    alertName
    location
    alertDescription
    alertSeverity
    isEnabled
    resourceId
    evaluationFrequency
    windowSize
    query
    metricMeasureColumn
    resourceIdColumn
    operator
    threshold
    timeAggregation
    numberOfEvaluationPeriods
    minFailingPeriodsToAlert
    muteActionsDuration
    autoMitigate
    checkWorkspaceAlertsStorageConfigured
    actionGroupId =
    $"""{{
        "type": "Microsoft.Insights/scheduledQueryRules",
        "apiVersion": "2021-08-01",
        "name": "{alertName}",
        "location": "{location}",
        "tags": {{}},
        "properties": {{
            "description": "{alertDescription}",
            "severity": "{alertSeverity}",
            "enabled": "{isEnabled}",
            "scopes": [
                "{resourceId}"
            ],
            "evaluationFrequency": "{evaluationFrequency}",
            "windowSize": "{windowSize}",
            "criteria": {{
                "allOf": [
                    {{
                        "query": "{query}",
                        "metricMeasureColumn": "{metricMeasureColumn |> Option.defaultValue ""}",
                        "resourceIdColumn": "{resourceIdColumn}",
                        "dimensions": [],
                        "operator": "{operator}",
                        "threshold": "{threshold}",
                        "timeAggregation": "{timeAggregation}",
                        "failingPeriods": {{
                            "numberOfEvaluationPeriods": "{numberOfEvaluationPeriods}",
                            "minFailingPeriodsToAlert": "{minFailingPeriodsToAlert}"
                        }}
                    }}
                ]
            }},
            "muteActionsDuration": "{muteActionsDuration}",
            "autoMitigate": "{autoMitigate}",
            "checkWorkspaceAlertsStorageConfigured": "{checkWorkspaceAlertsStorageConfigured}",
            "actions": {{
                "actionGroups": [
                    "{actionGroupId}"
                ],
                "customProperties": {{
                }}
            }}
        }}
    }}"""
    |> Resource.ofJson

let logDataAlert =
        scheduledQueryRule
            "Daily log data limit reached"
            "UKSouth"
            "Notify admins if log data limit reached"
            2
            "true"
            $"/subscriptions/{subId}/resourceGroups/{resGroupName}/providers/Microsoft.OperationalInsights/workspaces/{logAnalyticsName}"
            "PT5M"
            "PT5M"
            @"_LogOperation | where Category =~ 'Ingestion' | where Detail contains 'OverQuota'"
            None
            "_ResourceId"
            "GreaterThan"
            0
            "Count"
            1
            1
            "PT5M"
            "false"
            "false"
            alertAction.ActionGroupId

[Ryan-Palmer]

Here's a complete working sample.

It has Serilog structured logging, OTel metric and Traces plus the search query alert for data limits.

I will hopefully find time to make a proper Farmer PR rather than using the JSON, I just need to re-familiarise myself with the code, it's been a while!

https://github.com/Ryan-Palmer/Safe-Serilog-Farmer-Demo