Azure alerting tips

Logs are indispensable for troubleshooting but can also represent a significant cost, especially when using Azure's Application Insights and Log Analytics. A common mistake is (temporarily) enabling very extensive logging (e.g., Informational level) in production and forgetting to disable it. For one of our clients, this resulted in over a thousand euros in additional logging costs in a single month. 

Azure offers an option to set a hard limit on Log Analytics (e.g., a maximum of 2GB per day). The downside: once the quota is reached, logging stops abruptly. If something goes wrong after that, you miss crucial information.  

A smarter approach is to create an alert based on log volume. For example, set an alert to trigger if more than X gigabytes of logs are ingested per hour. That way you are alerted to abnormal spikes, you can investigate why (and adjust logging if necessary), but you don't lose data at a critical moment

To test the availability of your website, Azure recommends testing from five different locations to avoid false positives. Sounds good, but each additional test location incurs a cost: sometimes tens of euros extra per website, per week. For one of our clients, this amounted to an additional €160 per week! 

The solution? Scale back to three test locations, but make your alerting smarter. Instead of a simple alert like "if X locations fail", use log-based alerts (KQL) to implement incremental logic.  

For example, generate a P2 (lower priority) alert if one or two locations fail, but a P1 (high priority) alert if all three locations report an error. This way, you save on testing costs while maintaining a reliable picture of your actual availability. You will then only trigger the highest-urgency alert when genuinely necessary

This may seem like a detail, but a good naming convention for your alerts saves a huge amount of time during incidents. Make sure the name immediately makes it clear which resource, which customer and which condition is involved (e.g. [CustomerName]-[AppServiceName]-CPU > 90%). This way, you immediately know where to look in your email, ticket or monitoring dashboard. 

Also, use tags on your alerts. For example, tag the responsible team or the client's SPOC (Single Point of Contact). Every second counts during a P1 incident; you don't want to be wasting time identifying who to contact

The biggest enemy of effective alerting is alert fatigue: receiving so many (irrelevant) alerts that you start ignoring them. The result? When something genuinely serious is going on, you might miss it. 

The remedy: focus on quality over quantity. Don't just enable all the recommended alerts. Think critically: do I really need this alert? Is the threshold relevant to this particular environment? After all, standard thresholds are rarely perfect. Monitor new alerts closely and fine-tune them based on the actual performance of the environment. 

Also plan regular reviews (e.g. quarterly) to see which alerts often trigger, whether they are still relevant, and whether the thresholds need to be adjusted. It's better to have 5 well-configured, relevant alerts than 50 that only generate noise

Alerts are not just for technical issues. You should also set budget alerts on your Azure subscriptions or resource groups. This way, you get a notification when costs (or predicted costs) exceed a certain threshold. This helps you detect unexpected cost spikes early. 

Combine this with tagging, more specifically an owner tag on resource groups. If you receive a budget alert, the tag allows you to immediately identify who is responsible for those resources and ask them directly if everything is still required or if optimization is possible

Scheduled maintenance can cause an avalanche of alerts. To prevent your mailbox (or that of your on-call colleague) from being flooded, or to avoid unnecessary out-of-hours calls, use Alert Processing Rules.  

These allow you to temporarily suppress notifications for specific resources or action groups during a scheduled maintenance window (e.g. “no emails or calls between 20:00 and 22:00 for resource group X”). Important: the alerts themselves are still generated and visible in Azure afterwards, you only suppress the notifications

How often do you lose valuable time during an incident searching for the right documentation or solution? We recommend linking Knowledge Base (KB) articles directly to your alerts.  

Include the number or link of the relevant KB article (from Confluence, ServiceNow, or wherever you manage your documentation) in the alert's description or add it as a tag. If the alert then triggers, the engineer can immediately click through to the solution, without having to search. Ensure you have a well-structured, centralized knowledge base for each client or application

Effective alerting in Azure is a continuous process of setting up, monitoring, analyzing and optimizing. The most important lesson? Think before you act. Don't blindly follow recommendations, but weigh the pros and cons. Be critical of thresholds and relevance. Focus on quality and clarity. A well-thought-out alerting strategy will save you both money and valuable time, and will also prevent the dreaded alert fatigue.