Pre-Provisioned Grafana Alerts

Introduction

Every Mendix on Azure cluster is provisioned with a set of Grafana alert rules that monitor your apps, the underlying Kubernetes nodes, and the cluster database. These rules are supplied as examples. They cover common failure conditions and give you a working starting point, but they are not intended to be a complete monitoring solution.

You own these rules after they are created. Provisioning runs again whenever the cluster is updated, but a rule that already exists is left exactly as it is, so edits you make to thresholds, severities, or pending periods are never reverted. A rule you delete is treated as missing and is created again on the next cluster update. To stop a rule from evaluating permanently, pause it instead of deleting it.

Review the rules against your own availability targets and adjust them. The thresholds that ship with the cluster are generic defaults, and no default is correct for every app.

Viewing the Alert Rules

The rules are provisioned into the Mendix folder in Grafana, in two groups: Mendix Application Alerts for the application and node rules, and Mendix PostgreSQL Alerts for the database rules. Both groups evaluate every minute.

To view them, perform the following steps:

1. In Grafana, go to Alerting > Alert rules.
2. Expand the Mendix folder.
3. Click a rule name to see its query, condition, and current state.

Accessing Grafana requires the Grafana Admin permission on the target Azure subscription. For more information, see Monitoring Mendix on Azure with Grafana.

Application Alerts

These rules monitor the apps running in the mendix namespace. Metrics come from Azure Managed Prometheus.

The CPU and container memory alerts compare usage against the limits you set for the environment, not against the node capacity. If you raise an environment's memory limit, the alert adjusts automatically because the threshold is a percentage.

Deployment Has No Available Replicas is the most significant rule in the set, because it detects actual downtime rather than a warning sign. It fires when every replica of an app is unavailable. The rule ignores deployments younger than 10 minutes so that normal startup and rolling restarts do not trigger it.

Interpreting the Two Memory Alerts

Mendix apps run on a Java runtime, and the two memory alerts measure different things. Read them together rather than treating either one as the whole picture.

High Memory Usage measures the memory the container holds from the operating system's point of view. A Java runtime keeps memory it has already reclaimed internally instead of returning it to the operating system, so this figure rises to a plateau during normal operation and stays there. A high value on its own is not evidence of a memory problem. The threshold is 95% because that is the point at which the pod is close to being terminated for exceeding its limit.

Java Heap Memory Pressure measures how much of the old generation heap the app itself is using. This is the better signal for application memory problems. When it stays above 85% for 15 minutes, the app is holding nearly as much long-lived data as the heap allows, garbage collection runs more often, and its pauses last longer. Expect slower response times. Either raise the environment's memory limit or investigate what the app keeps in memory.

Node Alerts

These rules monitor the Azure Kubernetes Service nodes that host your apps.

Node pressure is often relieved automatically, because cluster autoscaling adds nodes when the existing ones cannot schedule new pods. Treat these alerts as capacity-planning signals rather than incidents that need an immediate response.

Database Alerts

These rules monitor the Azure Database for PostgreSQL flexible server instance. Metrics come from Azure Monitor rather than Prometheus.

PostgreSQL High Storage Usage (Auto-grow will trigger) has an informational severity because storage auto-grow is enabled. Azure expands the volume before it fills, so this alert is for cost and capacity awareness rather than an outage risk.

Configuring SMTP for Email Notifications

Each cluster is provisioned with a contact point named email-alerts, and a notification policy that routes all alerts to it. The email address is taken from the account that deployed the cluster. If that address cannot be determined, a placeholder is used instead, and you must set a real address before notifications can reach you.

Azure Managed Grafana does not include a mail server. You must supply your own SMTP relay before any notification can be delivered.

Setting the SMTP Configuration

To configure SMTP, run the following Azure CLI command, replacing the values with those of your own mail relay:

az grafana update \
  --resource-group <your-resource-group> \
  --name <your-grafana-instance> \
  --smtp true \
  --host "smtp.example.com:587" \
  --user "<smtp-username>" \
  --password "<smtp-password>" \
  --from-address "grafana-alerts@example.com" \
  --from-name "Mendix on Azure Alerts" \
  --start-tls-policy OpportunisticStartTLS

The following table describes the parameters that most often need attention.

Verifying Email Delivery

To confirm that the configuration works, perform the following steps:

1. In Grafana, go to Alerting > Contact points.
2. Click the edit icon for the email-alerts contact point.
3. Change the address in Addresses to your own if the deploying account's address is not the one you want to use.
4. Click Test and then click Send test notification.
5. Check the inbox of the recipient address.

If the test email does not arrive, verify that the relay accepts the sender address in --from-address, and that your STARTTLS policy matches what the relay expects.

An address you set here is replaced with the deploying account's address on the next cluster update. To send notifications to an address that persists, such as a team distribution list, create your own contact point and route to it from a notification policy of your own.

Notification Grouping

The provisioned notification policy groups notifications by alert name, namespace, and pod. Grafana waits 30 seconds before sending the first notification for a new group, waits 5 minutes before sending an update for a group that has already notified, and repeats an unresolved notification every 4 hours.

Adjusting the Alerts for Your Environment

Because these rules are examples, expect to change them. The following adjustments are the most common.

• Change a threshold - Open the rule in Alerting > Alert rules, edit the threshold expression, and save. An app that runs steadily at 85% CPU by design needs a higher threshold than the default.
• Change the pending period - Increase the pending period to reduce noise from short spikes, or decrease it to detect problems sooner.
• Route by severity - Add nested routes under your notification policy so that critical alerts reach an on-call channel while warnings go to email.
• Add a contact point - Grafana supports Slack, Microsoft Teams, PagerDuty, Opsgenie, and webhooks. Create the contact point, then reference it from a notification policy. Create a new contact point rather than editing email-alerts, because the provisioned one is reset on every cluster update.
• Disable a rule - Pause a rule to stop it evaluating. Pause rather than delete, because a deleted rule is created again on the next cluster update.

For the procedure to build an entirely new rule, see Setting up Grafana Alerts.

Alerts That Report No Data

An alert rule shows the No Data state when its query returns no series. This is normal in several situations, and does not always indicate a problem.

Read More

• Monitoring Mendix on Azure with Grafana - How to access Grafana and view logs and metrics
• Export Logs to External Systems - How to send logs to a third-party observability tool
• Support for Mendix on Azure - The responsibility split between Mendix and you
• Grafana Alerting documentation - Grafana's reference for alert rules, contact points, and notification policies
• Configure SMTP settings in Azure Managed Grafana - Microsoft's documentation for the SMTP options