Retrieve Environment-Sensitive Data from a Secret Store
1 Introduction
You can increase the security of your environment by implementing an external secrets store to manage your Kubernetes secrets. Environments running Mendix for Private Cloud can be granted read-only access to a secrets store by using a Kubernetes Secrets Store CSI Driver. This document outlines the high-level process, and provides example implementations for HashiCorp Vault and AWS Secrets Manager.
Using an external secret storage provides multiple benefits, such as rotating credentials from a single location, collecting audit logs and dynamically generating role-specific credentials.
Using a secret storage incorrectly may reduce the security of your app. This document describes a simplified approach to setting up Vault and should not be used for production environments. Consult with your secrets store provider to ensure that it is set up securely for your production environment.
1.1 Supported Stores
Mendix apps currently support the following secret stores:
- AWS Secrets Manager
- HashiCorp Vault
2 Configuring Your Environment
To implement an external secret store, you must configure the required settings by following these steps:
- Set up and configure the secret storage provider, for example, HashiCorp Vault, or AWS Secret Manager. In most cases, this should only be done once for the entire cluster. For more information and support, contact the secret storage provider.
- Install and configure a Kubernetes Secrets Store CSI driver, for example, AWS Secrets Manager CSI Secrets Store. This driver is installed globally for the entire cluster. For more information, refer to documentation supplied by the secret storage provider.
- Prepare a Kubernetes
ServiceAccount
to be used for authentication. TheServiceAccount
name must match the Mendix App CR name (that, is, the internal name of the app environment). In addition, theServiceAccount
needs to have aprivatecloud.mendix.com/environment-account: "true"
annotation. Your secret storage provider may have other requirements - for more information, refer to documentation supplied by the secret storage provider. Typically, the KubernetesServiceAccount
requires vendor-specific annotations to link it with an account or role in the secret storage provider. - Configure the SecretProviderClass
The
SecretProviderClass
contains vendor-specific configuration that specifies mapping rules - where to read the keys, how to transform and rename them. You must configure it to match the Mendix App CR name, and use a specific list of mappable keys - for more information, see SecretProviderClass Keys. In addition, theSecretProviderClass
should have aprivatecloud.mendix.com/environment-class: "true"
annotation. Your secret store provider may have other requirements - for more information, refer to documentation supplied by the secret store provider. - Configure permissions to read secrets specified in the
SecretProviderClass
. The specific configuration requirements depend on your secret store provider. It usually involves allowing the KubernetesServiceAccount
to access specific keys in Vault or in AWS console.
2.1 SecretProviderClass
Keys
The following table lists the properties used as keys for database and storage-related data. Use the following values when configuring the mapping rules in your SecretProviderClass
.
Data type | Key | Example | Required |
---|---|---|---|
Database type (for example, SQLSERVER, PostgreSQL) | database-type |
PostgreSQL |
✓ |
Database Jdbc Url | database-jdbc-url |
jdbc:postgresql://pg.example.com:5432/my-app-1?sslmode=prefer |
✓ |
Database Host | database-host |
pg.example.com:5432 |
✓ |
Database Name | database-name |
my-app-1 |
✓ |
Database Username | database-username |
my-app-user-1 |
✓ |
Database Password | database-password |
Welc0me! |
|
Storage service name | storage-service-name |
com.mendix.storage.s3 |
✓ |
S3 Storage endpoint | storage-endpoint |
https://my-app-bucket.s3.eu-west-1.amazonaws.com |
✓ (only for S3) |
S3 Storage access key id | storage-access-key-id |
AKIA################ |
|
S3 Storage secret access key | storage-secret-access-key |
A####################################### |
|
S3 subdirectory (or bucket name for S3-like storage systems) | storage-bucket-name |
subdirectory |
✓ (only for S3) |
Azure storage account | storage-azure-account-name |
example |
✓ (only for Azure Blob Storage) |
Azure storage account key | storage-azure-account-key |
aw== |
✓ (only for Azure Blob Storage) |
Azure storage container name | storage-azure-container |
examplecontainer |
✓ (only for Azure Blob Storage) |
Use HTTP for Azure | storage-azure-use-https |
true |
|
Use configured CA trust for file storage | storage-use-ca-certificates |
true |
|
Delete files from storage when deleted in the app | storage-perform-delete |
true |
|
Mendix Admin Password | mx-admin-password |
Welc0me! |
|
Mendix Debugger Password | mx-debugger-password |
Welc0me! |
|
App constant {name} |
mx-const-{name} |
mx-const-MyFirstModule.WelcomePageTitle |
|
Runtime custom setting {name} |
mx-runtime-{name} |
mx-runtime-com.mendix.storage.s3.EncryptionKeys |
storage-service-name
must specify one of the following supported blob storage providers:
com.mendix.storage.s3
for AWS S3 or S3-compatible providerscom.mendix.storage.azure
for Azure Blob Storage
storage-access-key-id
or storage-secret-access-key
to access an S3 bucket. Instead, you can use the same AWS IAM role for RDS authentication.
For more information and a complete walkthrough example, see Configuring a Secret Store with AWS Secrets Manager.
database-password
to access a Postgres RDS database. Instead, you can use the same AWS IAM role for RDS authentication.
For more information and a complete walkthrough example, see the AWS RDS IAM authentication example.
To set a Mendix app constant, use the mx-const-{name}
format (replace {name}
with the name of the app constant).
For example, if you need to set the MyFirstModule.WelcomePageTitle
constant, specify its value via the mx-const-MyFirstModule.WelcomePageTitle
key.
To set a Mendix Runtime custom setting, use the mx-runtime-{name}
format (replace {name}
with the name of the custom setting).
For example, if you need to set the com.mendix.storage.s3.EncryptionKeys
constant, specify its value via the mx-runtime-com.mendix.storage.s3.EncryptionKeys
key.
For a full configuration example, see Configuring a Secret Store with AWS Secrets Manager.
storage-azure-
prefix), upgrade to Mendix Operator version 2.10.0 or above.
mx-const-*
keys) and Mendix Runtime custom settings (mx-runtime-*
) from CSI Secrets Storage, upgrade to Mendix Operator version 2.10.0 or above.
mx-debugger-password
) from CSI Secrets Storage, upgrade to Mendix Operator version 2.15.0 or above.
3 Sample Implementations
The following sections outline the process of implementing an external secret store with Vault and with AWS. You can refer to them as an example, and to help you troubleshoot your own implementation.
3.1 Configuring a Secret Store with Vault
To enable your environment to use Vault as external secret storage, follow these steps:
-
Install Vault and its CSI Secrets Driver, if it is not already installed in the cluster.
-
Install CSI Secret Store Driver, as shown in the following example. Replace
<{ns}>
with the namespace name where Vault is installed:1 2 3
helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts helm -n <{ns}> install csi secrets-store-csi-driver/secrets-store-csi-driver \ --set syncSecret.enabled=true
-
Configure a Postgres or SQLServer database server with the following:
- A dedicated database to store your secrets
- An S3-compatible storage
-
Set up the secret in the database by starting an interactive shell session on the
vault-0
pod, as shown in the following example. Replace<{ns}>
with the namespace name where Vault is installed:1
kubectl -n <ns> exec -it vault-0 -- /bin/sh
-
Enable the Kubernetes authentication method by running the following command:
1
vault auth enable kubernetes
-
Configure the Kubernetes authentication method to use the service account token, the location of the Kubernetes host, and its certificate, as shown in the following example:
1 2 3
vault write auth/kubernetes/config \ kubernetes_host=https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT \ kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
-
Create a database secret in Vault, as shown in the following example. Replace
<{env-db-secret}>
with a unique name:1
vault kv put secret/<{env-db-secret}> database-type="PostgreSQL" database-jdbc-url="jdbc:postgresql://pg.example.com:5432/my-app-1?sslmode=prefer" database-host="pg.example.com:5432" database-name="my-app-1" database-username="my-app-user-1" database-password="Welc0me!"
-
Create a file storage secret in Vault, as shown in the following example. Replace
<{env-file-secret}>
with a unique name, and update any values to match your file storage configuration:1
vault kv put secret/<{env-file-secret}> storage-service-name="com.mendix.storage.s3" storage-endpoint="https://my-app-bucket.s3.eu-west-1.amazonaws.com" storage-access-key-id="AKIA################" storage-secret-access-key="A#######################################" storage-bucket-name="subdirectory" storage-use-ca-certificates="true" storage-perform-delete="true"
-
Create an app environment configuration secret in Vault, as shown in the following example. Replace
<{env-configuration-secret}>
with a unique name, and set any additional parameters:1
vault kv put secret/<{env-configuration-secret}> mx-admin-password="Welc0me!" mx-debugger-password="Welc0me!"
-
Create the required Vault role, as shown in the following example. Replace
<{env-policy}>
with a unique name to identify the app environment, and update any paths to match the secrets you created in the previous steps:1 2 3 4 5 6 7 8 9 10 11
vault policy write <{env-policy}> - <<EOF path "secret/data/<{env-db-secret}>" { capabilities = ["read"] } path "secret/data/<{env-file-secret}>" { capabilities = ["read"] } path "secret/data/<{env-configuration-secret}>" { capabilities = ["read"] } EOF
-
Bind the Vault role to a Kubernetes service, as shown in the following example. Replace
<{env-policy}>
with the policy name from the previous step, use a unique role name in place of<{env-role}>
, and specify the environment’s Kubernetes namespace and ServiceAccount in place of<{env-namespace}>
and<{env-serviceaccount}>
):1 2 3 4
vault write auth/kubernetes/role/<{env-role}> \ bound_service_account_names=<{env-serviceaccount}> \ bound_service_account_namespaces=<{env-namespace}> \ policies=<{env-policy}>
-
Create a Kubernetes
ServiceAccount
for your environment, as shown in the following example. Specify the environment’s Kubernetes namespace and ServiceAccount in place of<{env-namespace}>
and<{env-serviceaccount}>
:1 2
kubectl -n <{env-namespace}> create serviceaccount <{env-serviceaccount}> kubectl -n <{env-namespace}> annotate serviceaccount <{env-serviceaccount}> privatecloud.mendix.com/environment-account=true
-
Create the
SecretProviderClass
CR for the Secrets Store CSI Driver:1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65
apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: <{MendixApp CR name}> annotations: privatecloud.mendix.com/environment-class: "true" spec: provider: vault parameters: vaultAddress: "http://vault.{<Vault-namespace>}.svc.cluster.local:8200" roleName: "<{env-role}>" objects: | - secretKey: "database-type" objectName: "database-type" secretPath: "secret/data/<{env-db-secret}>" - secretKey: "database-jdbc-url" objectName: "database-jdbc-url" secretPath: "secret/data/<{env-db-secret}>" - secretKey: "database-username" objectName: "database-username" secretPath: "secret/data/<{env-db-secret}>" - secretKey: "database-password" objectName: "database-password" secretPath: "secret/data/<{env-db-secret}>" - secretKey: "database-host" objectName: "database-host" secretPath: "secret/data/<{env-db-secret}>" - secretKey: "database-name" objectName: "database-name" secretPath: "secret/data/<{env-db-secret}>" - secretKey: "storage-service-name" objectName: "storage-service-name" secretPath: "secret/data/<{env-file-secret}>" - secretKey: "storage-endpoint" objectName: "storage-endpoint" secretPath: "secret/data/<{env-file-secret}>" - secretKey: "storage-access-key-id" objectName: "storage-access-key-id" secretPath: "secret/data/<{env-file-secret}>" - secretKey: "storage-secret-access-key" objectName: "storage-secret-access-key" secretPath: "secret/data/<{env-file-secret}>" - secretKey: "storage-bucket-name" objectName: "storage-bucket-name" secretPath: "secret/data/<{env-file-secret}>" - secretKey: "storage-perform-delete" objectName: "storage-perform-delete" secretPath: "secret/data/<{env-file-secret}>" - secretKey: "storage-use-ca-certificates" objectName: "storage-use-ca-certificates" secretPath: "secret/data/<{env-file-secret}>" - secretKey: "mx-admin-password" objectName: "mx-admin-password" secretPath: "secret/data/<{env-configuration-secret}>" - secretKey: "mx-debugger-password" objectName: "mx-debugger-password" secretPath: "secret/data/<{env-configuration-secret}>" # Example: use MyFirstModule.MyConstant constant value from AWS Secrets Manager #- secretKey: "MyFirstModule.MyConstant" # objectName: "mx-const-MyFirstModule.MyConstant" # secretPath: "secret/data/<{env-configuration-secret}>" # Example: use com.mendix.storage.s3.EncryptionKeys custom setting from AWS Secrets Manager #- secretKey: "com.mendix.storage.s3.EncryptionKeys" # objectName: "mx-runtime-com.mendix.storage.s3.EncryptionKeys" # secretPath: "secret/data/<{env-configuration-secret}>"
-
Create an app with the secret store enabled. If you are using the Portal, secret stores are enabled automatically if the Enable Secrets Store option is activated for the namespace where you create the app. For a standalone app, you must set the value of the
allowOverrideSecretsWithSecretStoreCSIDriver
setting totrue
in the Mendix app CRD. The following yaml shows an example Mendix app CRD:1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
cat > mendixApp.yaml <<EOF apiVersion: privatecloud.mendix.com/v1alpha1 kind: MendixApp metadata: name: <{Mendix App CR name}> spec: mendixRuntimeVersion: 9.4.0.24572 allowOverrideSecretsWithSecretStoreCSIDriver: true replicas: 1 resources: limits: cpu: "1" memory: 512Mi requests: cpu: 100m memory: 512Mi runtime: customConfiguration: '{"ScheduledEventExecution":"NONE","MicroflowConstants":"{\"MyFirstModule.MyConstant\":\"Awesome\",\"RestClient.RestServiceUrl\":\"https://go-dummy-app.privatecloud-storage-tls.svc.cluster.local\",\"Atlas_Core.Atlas_Core_Version\":\"3.0.5\"}"}' dtapMode: D logAutosubscribeLevel: INFO runtimeLicense: {} runtimeMetricsConfiguration: {} sourceURL: oci-image://<{image URL}> sourceVersion: 0.0.0.87 EOF
kv-v2
keystore, paths should be prefixed with secrets/data/
. Please refer to the Hashicorp Vault documentation for more information.
3.2 Configuring a Secret Store with AWS Secrets Manager
To enable your environment to use AWS Secrets Manager as external secret storage, follow these steps:
-
In AWS Secrets Manager, create a new secret of the type Other.
-
Define the keys as described in the SecretProviderClass Keys section above, and then click Next.
-
Enter the secret name, for example,
<namespace>/<Kubernetes environment name>
. -
Follow the wizard to complete the secret creation process.
-
Open the secret and copy the Secret name and Secret ARN.
-
In the Cloud Portal, create an environment using secrets storage, with no database or storage plan.
-
Create an IAM Role without any attached policies for that environment in the AWS console.
Use the environment internal name as the service account name. -
In the IAM role, add an inline policy with the following JSON:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret" ], "Resource": [ "<{The secret ARN that you copied in step 5 above}>" ] } ] }
-
Allow a Kubernetes ServiceAccount to assume a role.
-
Open the role for editing and add an entry for the ServiceAccount (or ServiceAccounts) to the list of conditions:
-
For the second condition, copy and paste the
sts.amazonaws.com
line; replace:aud
with:sub
and set it tosystem:serviceaccount:<Kubernetes namespace>:<Kubernetes serviceaccount name>
.See Amazon EKS Pod Identity Webhook – EKS Walkthrough for more details.
The role ARN is required, you can use the Copy button next to the ARN name in the role details.
After this, the specified serviceaccount in the specified namespace will be able to assume this role.
-
-
Create a Kubernetes
ServiceAccount
for your environment:1 2 3
kubectl -n <{Kubernetes namespace}> create serviceaccount <{environment name}> kubectl -n <{Kubernetes namespace}> annotate serviceaccount <{environment name}> privatecloud.mendix.com/environment-account=true kubectl -n <{Kubernetes namespace}> annotate serviceaccount <{environment name}> eks.amazonaws.com/role-arn=<{aws role ARN}>
-
Create an app with the secret store enabled. If you are using connected mode, secret stores are enabled automatically if the Enable Secrets Store option is activated for the namespace where you create the app. For a standalone app, you must set the value of the setting
allowOverrideSecretsWithSecretStoreCSIDriver
totrue
in the Mendix app CRD.The following yaml shows an example Mendix app CRD:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
cat > mendixApp.yaml <<EOF apiVersion: privatecloud.mendix.com/v1alpha1 kind: MendixApp metadata: name: <{Mendix App CR name}> spec: mendixRuntimeVersion: 9.4.0.24572 allowOverrideSecretsWithSecretStoreCSIDriver: true replicas: 1 resources: limits: cpu: "1" memory: 512Mi requests: cpu: 100m memory: 512Mi runtime: customConfiguration: '{"ScheduledEventExecution":"NONE","MicroflowConstants":"{\"MyFirstModule.MyConstant\":\"Awesome\",\"RestClient.RestServiceUrl\":\"https://go-dummy-app.privatecloud-storage-tls.svc.cluster.local\",\"Atlas_Core.Atlas_Core_Version\":\"3.0.5\"}"}' dtapMode: D logAutosubscribeLevel: INFO runtimeLicense: {} runtimeMetricsConfiguration: {} sourceURL: oci-image://<{image URL}> sourceVersion: 0.0.0.87 EOF
-
Attach the secret to the environment by applying the following Kubernetes yaml:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49
apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: <{MendixApp CR name}> annotations: privatecloud.mendix.com/environment-class: "true" spec: provider: aws parameters: objects: | - objectName: "<{The secret name that you copied in step 5 above}>" objectType: "secretsmanager" jmesPath: - path: '"database-type"' objectAlias: "database-type" - path: '"database-jdbc-url"' objectAlias: "database-jdbc-url" - path: '"database-username"' objectAlias: "database-username" - path: '"database-password"' objectAlias: "database-password" - path: '"database-host"' objectAlias: "database-host" - path: '"database-name"' objectAlias: "database-name" - path: '"storage-service-name"' objectAlias: "storage-service-name" - path: '"storage-endpoint"' objectAlias: "storage-endpoint" - path: '"storage-access-key-id"' objectAlias: "storage-access-key-id" - path: '"storage-secret-access-key"' objectAlias: "storage-secret-access-key" - path: '"storage-bucket-name"' objectAlias: "storage-bucket-name" - path: '"storage-perform-delete"' objectAlias: "storage-perform-delete" - path: '"storage-use-ca-certificates"' objectAlias: "storage-use-ca-certificates" - path: '"mx-admin-password"' objectAlias: "mx-admin-password" - path: '"mx-debugger-password"' objectAlias: "mx-debugger-password" # Example: use MyFirstModule.MyConstant constant value from AWS Secrets Manager #- path: '"MyFirstModule.MyConstant"' # objectAlias: "mx-const-MyFirstModule.MyConstant" # Example: use com.mendix.storage.s3.EncryptionKeys custom setting from AWS Secrets Manager #- path: '"com.mendix.storage.s3.EncryptionKeys"' # objectAlias: "mx-runtime-com.mendix.storage.s3.EncryptionKeys"
In the above example, path
specifies the key name from the original AWS Secret (Secret Manager key), and objectAlias
specifies how it will be named when mounted into the sidecar. Do not modify objectAlias
, as it matches the AWS objectName
.
If your app is created in Mendix 9.20 or above, you can remove storage-access-key-id
and storage-secret-access-key
parameters, and instead attach this IAM policy to the app’s IAM role (replace <bucket_name>
with the name of the S3 bucket):
|
|
This means that the app authenticates with the AWS S3 API using AWS IRSA instead of static credentials.
3.3 Using IAM authentication for AWS RDS databases
AWS RDS Postgres databases can use IAM database authentication instead of regular passwords.
To use this feature, you need to:
- Use an AWS RDS Postgres database with IAM authentication enabled.
- Use Mendix Operator version 2.10.1 and above.
- Use Mendix 9.22 and above.
- Complete the steps described in Configuring a Secret Store with AWS Secrets Manager.
After completing the prerequisites, follow these steps to switch from password-based authentication to IAM authentication:
-
Remove or comment out
database-password
from theSecretProviderClass
and the associated AWS Secret. -
Enable IAM authentication for the
database-username
role by using thepsql
commandline to run the following commands (replacing<database-username>
with the username specified indatabase-username
):GRANT rds_iam TO <database-username>; ALTER ROLE <database-username> WITH PASSWORD NULL;
This step is not necessary if the RDS instance was created with only IAM authentication enabled, and ifdatabase-username
is the default (primary) user. -
Attach the following inline IAM policy to the environment’s IAM role (created when Configuring a Secret Store with AWS Secrets Manager):
1 2 3 4 5 6 7 8 9 10 11 12 13 14
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "rds-db:connect" ], "Resource": [ "arn:aws:rds-db:<db-region>:<account-id>:dbuser:<db-resource-id>/<database-username>" ] } ] }
replacing
<db-region>
with the RDS database region,<account-id>
with the AWS account ID,<db-resource-id>
with the database Resource ID and<database-username>
with the username specified indatabase-username
.For more information on how to get the Resource ID and create an RDS IAM policy, see the AWS documentation.
-
Restart the Mendix app environment.
When using IAM authentication, the Mendix app’s environment (m2ee-sidecar
container) uses that app’s attached IAM role to request a new Postgres password every 10 minutes from the RDS API. These passwords expire after 15 minutes.
Passwords are only checked when opening a new connection, so an expired password does not cancel any existing connections or interrupt any running database transactions and queries.
4 Additional considerations
When implementing a secret store, keep in mind the following considerations:
- It is not currently possible to use Storage Plans with CSI Secrets Storage. Instead, your infrastructure admin must attach a Kubernetes
ServiceAccount
andSecretProviderClass
would be attached to the app before or after the environment is created. - If a secret is rotated or updated, Mendix Operator version 2.10.0 and above will detect the changes and apply them. However (depending on the Mendix Runtime version in use), the changes might not be applied. If the changes are not applied, you should restart the environment. Bear in mind the following features of detecting a changed secret:
- There is a delay of a few minutes before the CSI Secrets Storage driver detects the changes.
- Only file storage credentials and
MxAdmin
password changes are correctly processed at the moment. - In Mendix 9.22 or above, database password rotation is processed without restarting the app.
- Dynamic secrets in HashiCorp Vault are supported - from the app environment, they are identical to regular secrets.
- The internal name of the environment must match an existing
ServiceAccount
andSecretProviderClass
. - CSI Secrets Storage can override app settings — if a parameter is configured in the Developer Portal or
MendixApp
CR, the value from CSI Secrets Storage will have a higher priority and will override the value specified elsewhere. For example, CSI Secrets Storage can override theMxAdmin
password, app constants, and runtime custom settings.
Feedback
Was this page helpful?
Glad to hear it! Thank you for your response.
Sorry to hear that. Please tell us how we can improve.