Azure Key Vault Integration

Introduction

Private Mendix Platform can use Azure Key Vault to automatically manage sensitive, environment-specific configuration, such as app constants, Mendix Runtime custom settings, the MxAdmin password, and the Debugger password, instead of storing them as plain values in the database.

The integration uses the Kubernetes Secrets Store CSI Driver with the Azure Key Vault provider. Every environment must have a User-Assigned Managed Identity (UAMI) from Azure Workload Identity. Private Mendix Platform automates the Kubernetes-side configuration of the ServiceAccount and SecretProviderClass objects and the Key Vault writes. The customer only needs to manage the Azure-side infrastructure (such as Key Vaults, managed identities, and RBAC), instead of manually configuring Kubernetes manifests for every environment.

Secret values are never stored in the Private Mendix Platform database or converted to a Kubernetes Secret object. They exist in Azure Key Vault, and are mounted into the running app pod as in-memory (tmpfs) files by the CSI driver.

Prerequisites

The customer is responsible for configuring the following prerequisites. They must be in place once per cluster, before any environment can use this feature.

• Your cluster type must be set to AKS. To confirm this, go to Switch to Admin Mode > Manage > Cluster Manager, and edit or create a cluster. On the Cluster Information page, verify that the Cluster Type field shows AKS. Other cluster types do not currently support Azure Key Vault integration.

• You must map the following supported secret types 1-to-1 to entries in the environment's SecretProviderClass. Other secret types are not supported.

  • App constant - This secret type is configured as a constant name and value in the Private Mendix Platform UI. It represents a Mendix app constant, for example, MyFirstModule.WelcomePageTitle.
  • Runtime custom setting - This secret type is configured as a setting name and value in the Private Mendix Platform UI. It represents a Mendix Runtime custom setting for m2ee configuration.
  • MxAdmin Password - This secret type is a fixed value, configured once per environment. It represents the password of the environment's administrator account.
  • Debugger Password - This secret type is a fixed value, configured once per environment. It represents the password of the environment's remote debugger.

• You must install the Secrets Store CSI Driver and Azure Key Vault provider cluster-wide (typically in kube-system), by using the following Helm commands:

  helm repo add csi-secrets-store-provider-azure https://azure.github.io/secrets-store-csi-driver-provider-azure/charts
  helm install csi-secrets-store-provider-azure csi-secrets-store-provider-azure/csi-secrets-store-provider-azure -n kube-system

  This chart bundles the generic secrets-store-csi-driver as a sub-chart dependency. Installing it also installs the underlying driver.

• The Azure Workload Identity webhook must be enabled and active on the cluster.

• OIDC issuer must be enabled on the AKS cluster. This is required for federated credentials on managed identities.

• Kubernetes RBAC must be configured as described in Kubernetes RBAC Configuration.

Kubernetes RBAC Configuration

Private Mendix Platform authenticates to the Kubernetes API using a bearer token (a dedicated ServiceAccount, for example, mxplatform-cicd), distinct from the app-environment identities. For information about configuring the bearer token, see Private Mendix Platform Functionalities: Cluster Manager.

This token needs a namespaced Role and RoleBinding in every namespace where app environments using this feature are deployed, granting the following:

Verify the configuration by using the following command:

kubectl auth can-i create secretproviderclasses --as=system:serviceaccount:kube-system:mxplatform-cicd -n <namespace>

Example Manifest

The following manifest is an example showing how to create roles and role bindings for each namespace where you wish to deploy apps using the automatic Key Vault feature.

Key Vault permissions for app namespace ns-apps-1 (Role + RoleBinding pointing to mxplatform-cicd SA)
NS=ns-apps-1
kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pmp-secret-management
  namespace: ${NS}
rules:
- apiGroups: ["secrets-store.csi.x-k8s.io"]
  resources: ["secretproviderclasses"]
  verbs: ["get","create","patch","delete"]
- apiGroups: [""]
  resources: ["serviceaccounts"]
  verbs: ["get","create","patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pmp-secret-management-binding
  namespace: ${NS}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: pmp-secret-management
subjects:
- kind: ServiceAccount
  name: mxplatform-cicd
  namespace: kube-system
EOF

Known Limitations

The following limitations apply:

• In the current version of Private Mendix Platform, Azure Key Vault integration can only be enabled during environment creation. It cannot be enabled for existing environments.

• Database and blob storage connection secrets are not supported through this automatic flow. You must provision them manually. For more information, refer to the general Secret Store documentation.

• Azure-wi storage and database plans are not compatible with Interactor-Agent deployments that also use the external secret store. The azure.workload.identity/client-id annotation on the ServiceAccount can only hold one value at a time, and the Operator may overwrite it. This feature is supported only in Standalone mode.

• An environment can use either a SecretProviderClass or a regular Kubernetes Secret for its configuration, never both at the same time.

• Secret store cannot be enabled for the following custom properties:

  • Custom environment variables
  • Custom JVM options
  • Custom Jetty options

Configuring the Integration

To enable the integration, you must first enable secret store support for the namespace where you want to create an environment with Azure Key Vault integration. To do this, perform the following steps:

1. Log in to Private Mendix Platform as a user with company administrator access rights.

2. Switch to Admin Mode by clicking the profile picture in the top right corner of the screen and selecting Switch to Admin Mode.

3. In the left navigation menu, open the Manage section and go to Deployment > Cluster Manager.

4. Select or create a cluster with the Cluster Type set to AKS.

5. Edit or create the target namespace and toggle Enable Secret Store.

6. Select one of the following options:

   • Automatic mode - This mode is available only for clusters with the Cluster Type set to AKS. Automatic mode only supports Azure Key Vault integration.

     • If the environment uses azure-wi storage plans, the Mendix Operator creates a user-assigned managed identity and service account for the environment. You must create a Key Vault and grant it the relevant permissions for this managed identity.
     • If the environment does not use azure-wi storage plans, the Mendix Operator does not create a user-assigned managed identity and service account for the environment. You must create a user-assigned managed identity and a Key Vault with the relevant permissions for this managed identity. Private Mendix Platform will use this information to create the SecretProviderClass and the service account.

   • Manual mode - This is a legacy mode where the customer creates manages the CSI driver integration, the SecretProviderClass, the Key Vault, the mappings and the permissions for Azure workload identities, AWS IRSA, or Hashicorp vault. Manual mode can be used to configure secret stores other than Azure Key Vault.

After enabling the secret store support for the namespace, proceed by creating the environment and configuring the necessary settings. The specific steps are slightly different depending on whether your environment will use an Azure Workload Identity (azure-wi) storage or database plan. Private Mendix Platform automatically determines the correct path. When you enable the secret store for an environment, Private Mendix Platform checks whether a Kubernetes ServiceAccount already exists for that environment:

• If the ServiceAccount exists, Private Mendix Platform reads the UAMI Client ID directly from the ServiceAccount's existing annotation. No customer input is required.

• If the ServiceAccount does not exist, Private Mendix Platform shows a message like the following:

  Private Mendix Platform has detected that an auto-provisioned managed identity does not exist for this environment. In order to use the automatic secret store, request your infrastructure team to provision an Azure Managed Identity for this environment and provide the Client ID.

  The customer's infrastructure team must then create the UAMI in Azure and supply its Client ID in this prompt. Private Mendix Platform stores it and creates the Kubernetes ServiceAccount with the correct name and annotations. The customer does not need to create or annotate the service account manually.

For more details, refer to the sections below.

Approach A - With Azure Workload Identity Storage

For environments which will use an azure-wi storage or database plan, the managed identity (UAMI) is automatically created by the Mendix Operator during environment provisioning. The Kubernetes ServiceAccount is created by the Operator and annotated with the UAMI Client ID. Because of that, the customer must only configure Key Vault and RBAC.

The configuration requires the following steps:

1. The customer (portal user) or Mendix Operator creates the environment in the Private Mendix Platform wizard, selecting the following options:

   • Cluster and Namespace - Select the cluster and namespace where you previously enabled secret store support.

   • Database Plan and Storage Plan - Select an azure-wi database and/or storage plan.

   • Secrets - Select the checkbox Use external secret store for environment configurations.

     

   The Mendix Operator automatically creates the UAMI, its federated credential, and annotates the environment's ServiceAccount with the Client ID.

2. The customer's Infrastructure Team creates an empty Key Vault named mendix-<environment internal name>, with Azure RBAC selected as the permission model. For more information, see Naming Conventions and Kubernetes Object Reference.

3. The customer's Infrastructure Team grants the Key Vault Secrets Officer role on that Key Vault to the Private Mendix Platform managed identity.

4. The customer's Infrastructure Team grants the Key Vault Secrets User role on that Key Vault to the environment's UAMI.

No manual Kubernetes object creation or Client ID entry is needed for this path.

Approach B - Without Azure Workload Identity Storage

For environments which will not use azure-wi plans, the managed identity (UAMI) must be manually created by the customer before enabling Key Vault. Since the Kubernetes ServiceAccount does not exist, Private Mendix Platform detects that it is missing and asks for the Client ID, and then created the ServiceAccount. This approach requires the customer to configure the UAMI, in addition to Key Vault and RBAC.

The configuration requires the following steps:

1. The customer (portal user) creates the environment in the Private Mendix Platform wizard, selecting the following options:

   • Cluster and Namespace - Select the cluster and namespace where you previously enabled secret store support.

   • Database Plan and Storage Plan - Select a database and/or storage plan without azure-wi.

   • Secrets - Select the checkbox Use external secret store for environment configurations.

     

2. The customer's Infrastructure Team creates a User-Assigned Managed Identity for this environment in Azure. The recommended naming for the UAMI is mendix-<namespace><environment internal name><app CR UID>. This matches the convention the Operator itself uses in Approach A, for consistency.

3. The customer's Infrastructure Team adds a federated credential to the UAMI, where the issuer is the cluster's OIDC issuer URL, and the subject is system:serviceaccount:<namespace>:<environment internal name>.

4. Create an empty Key Vault named mendix-<environment internal name>, with Azure RBAC selected as the permission model.

5. The customer's Infrastructure Team grants the Key Vault Secrets Officer role on that Key Vault to the Private Mendix Platform managed identity.

6. The customer's Infrastructure Team grants the Key Vault Secrets User role on that Key Vault to the environment's UAMI.

7. The customer (portal user) enters the UAMI's Client ID created in step 2 above. Private Mendix Platform creates the Kubernetes ServiceAccount for the environment with both required annotations, and stores the Client ID for use when secrets are written.

You do not need to manually run any kubectl create serviceaccount or kubectl annotate commands. Private Mendix Platform performs this step after the user has entered the Client ID.

Reference

Refer to the following sections for more information about the required permissions, naming conventions, and other concerns.

Permissions Reference for Managed Identities and Azure RBAC

All of the following identities must be correctly configured, each with a different Azure role, a different scope, and a different party responsible for granting it. Incorrectly configuring any of the identities will cause errors with the integration.

Private Mendix Platform Managed Identity

The PMP-KeyVault-Identity is the Workload Identity assigned to the Private Mendix Platform pod (not to any customer environment). Private Mendix Platform uses it through DefaultAzureCredentialBuilder for every write, update, delete, and recover operation against a customer's Key Vault.

Environment-Level UAMI

This is the managed identity tied to a specific environment. It is created automatically by the Operator in Approach A, or manually by the customer in Approach B. It is used exclusively at runtime by the CSI driver running inside the app pod. Private Mendix Platform code never uses this identity.

Mendix Operator Provisioning Identity

This identity not a Key Vault access identity. It allows the Mendix Operator to automatically create the UAMI and connect it to an azure-wi storage or database plan. It is required for Approach A to work.

Naming Conventions and Kubernetes Object Reference

Secret Key Naming

Secret key naming inside the Key Vault or SecretProviderClass follows the general. Mendix conventions. For more information, see Retrieve Environment-Sensitive Data from a Secret Store.

Disallowed Characters for Azure Key Vault Secret Names

Azure Key Vault secret names cannot contain underscores (_), periods (.), spaces, and special symbols. Private Mendix Platform converts any periods in a constant or runtime setting name to hyphens for the Key Vault entry (objectName). The objectAlias - that is, the file name actually mounted into the pod and read by the Mendix Runtime - always keeps the periods, exactly matching the format in the official documentation, for example, mx-const-MyFirstModule.WelcomePageTitle. Private Mendix Platform performs this conversion consistently both when it writes the value to Key Vault and when it builds the SecretProviderClass entry, so the two always stay in sync. However, other special characters are not automatically converted, so constant names that contain them cannot be stored in Azure Key Vault. Because of that, we recommend that you do not create secret names with underscores (_), periods (.), spaces, and special symbols.

Additional Concerns

Key Vault Access Model

The Key Vault's access configuration must use the Azure RBAC permission model, not the legacy Vault access policies model. Role assignments like Secrets Officer or Secrets User only take effect under the RBAC model.

Tenant ID Source

The directory tenant ID used in the SecretProviderClass is found on the Key Vault's Overview page in the Azure Portal - it is the Azure AD tenant the Key Vault belongs to, not anything environment-specific.

Soft-Delete and Recovery

Azure Key Vault has soft-delete enabled by default and it cannot be disabled. Private Mendix Platform relies on this - when a secret name that was previously deleted is written again, Private Mendix Platform automatically recovers the soft-deleted secret before writing the new value, rather than failing. If a secret is ever purged (permanently deleted) directly in Azure, this automatic recovery is no longer possible and requires manual intervention.

Resource Providers

Microsoft.KeyVault and Microsoft.ManagedIdentity must be registered on the subscription.

Using Key Vault Integration

The following sections provide information about using the Key Vault integration for business-as-usual operations.

Adding New Values

To add a new app constant, runtime setting, MxAdmin password, or debugger password, log in to the Private Mendix Platform portal, and enter the value and optionally the name (for constants or runtime settings). The value is only held in memory while the request is processed. It is never written to the Private Mendix Platform database.

Private Mendix Platform writes the value to the environment's Key Vault under the correct name. If the Key Vault previously had this same secret name soft-deleted, Private Mendix Platform recovers it automatically before writing.

Private Mendix Platform checks whether the environment's SecretProviderClass already exists. If it does not, Private Mendix Platform creates it, with this secret as its first entry. The SecretProviderClass is never created empty.

If the SecretProviderClass already exists, Private Mendix Platform patches it in place, adding the new entry without modifying any existing ones.

Private Mendix Platform then shows a notification that the value was written but will not take effect until the environment is restarted.

Updating an Existing Value

Only the Key Vault value is updated. The SecretProviderClass is unchanged, because the mapping of key names to Key Vault entries has not changed. A restart is required for the new value to take effect in the running app. For already-mounted secrets, the CSI driver also automatically refreshes mounted values within approximately 2 minutes without requiring a restart, but restarting is the reliable way to guarantee it.

Deleting a Value

Deletion happens in two phases, always in this order:

1. The entry is removed from the SecretProviderClass on Kubernetes side.
2. The secret is soft-deleted from Azure Key Vault.

This order prevents a pod restart from failing to find a secret that the SecretProviderClass still references but which has already been removed from the vault.

Restarting the Environment

Every write, update, or delete requires an environment restart to take effect for a currently running pod, except for value-only updates to already-mounted secrets, which the CSI driver automatically picks up within approximately 2 minutes.

On the first ever write for an environment, Private Mendix Platform sets allowOverrideSecretsWithSecretStoreCSIDriver: true on the MendixApp CR. This happens only once, at restart time, not immediately at write time, in order to avoid the Operator reconciling against a pod that has not actually restarted with the new mount yet.

Troubleshooting

If you encounter any issues with Key Vault integration, use the following troubleshooting tips to help you solve them.

403 Forbidden Error

You get an 403 Forbidden error on serviceaccounts or secretproviderclasses.

Cause

The Private Mendix Platform cluster token does not have a namespace RBAC.

Solution

Apply and verify the Role and RoleBinding in the target namespace. Confirm by running the following command: kubectl auth can-i.

Missing Secretproviderclasses Resource Type

You see an error like the following: The server doesn't have a resource type secretproviderclasses.

Cause

The CSI Secrets Store Driver or Azure provider is not installed in the cluster.

Solution

Install the dependencies through the Helm chart.

CreateContainerError after Restart

After a restart, the pod fails to start and shows a CreateContainerError.

Cause

A secret listed in the SecretProviderClass does not exist in Key Vault (for example, it was deleted directly in Azure, bypassing Private Mendix Platform).

Solution

Add the missing secret again through Private Mendix Platform, or manually restore it in Key Vault. Never edit const, runtime, or password secrets directly in Azure. Always perform the changes through Private Mendix Platform.

Managed Identity Does Not Exist Error

The environment shows an error like Managed identity does not exist.

Cause

The ServiceAccount does not exist, or the Client ID entered does not correspond to a UAMI with a working federated credential.

Solution

Check the UAMI and its federated credential (issuer and subject).

Value Changes Not Showing

A new or updated value does not appear in the running app.

Cause

The environment was not restarted after the change.

Solution

Restart the environment from the Private Mendix Platform portal.

Wrong Azure Tenant or Access Fail

The pod authenticates to the wrong Azure tenant, or Key Vault access fails despite correct RBAC.

Cause

The Key Vault access configuration is set to Vault access policies" instead of Azure RBAC.

Solution

Recreate or reconfigure the Key Vault to use the Azure RBAC permission model.

Customer Responsibilities Summary