Advanced Audit Trail

I Introduction

Advanced Audit Trail allows you to trace changes and use infinitely-scalable and fully-indexed data search. Once configured, the system automatically creates audit snapshots of objects to store in the audit trail. The audit trail is centralized and sent to long-term data storage, therefore supporting complex search queries and keeping the operational database small and performant.

Advanced Audit Trail employs a software stack on top of Kafka, Elasticsearch, and Kibana to leverage their utility. Kafka is used for long-term immutable data storage for audit data. Elasticserach and Kibana are used to index audit data. You can find an integration diagram below:

1.1 Typical Use Cases

• Tracks changes to data (who made what changes at what time in what context) to make your operation GxP and FDA CFR 21 Part 11 compliant, which is required for medical interaction platforms as well as for processes such as credit transactions and internal IT audits for user access
• Allows the developer to easily configure tracking changes per entity, with no changes to domain models required
• Helps the developer debug why an object is in a specific stage

1.2 Features

• Records the timestamp, the old value, the changed value, the microflow that triggered the change, whether the object was created, modified, or deleted, the user who made the change, and their user role
• Supports viewing all changes that happened in the same microflow, which helps the auditor understand the context better
• Captures the checksum, file size, and name of files that can be used for validation
• Supports scheduled events that regularly send the stored snapshots to an external system
• Allows the developer to configure an input field where additional information can be supplied about the snapshot to be created—once added, this category will be automatically added to all snapshots
• Offers microflows and pages that open a generalized view to show users the trail of a specific object
• Supports decoupling—when the external system cannot be reached, the snapshots will be stored in the local database, thus ensuring that the main system will keep on working without a dependency on the external database
• Offers auditor interface to search through the external database (across entities)
• Supports full-text search on data and search on changed data, and also export of data to the CSV format using Kibana
• Support configuring different permissions for audit data for different users

1.3 Limitations

• Advanced Audit Trail contains an add-on module.
• On Mendix Cloud XS resource packs, your app may run out of memory due to too many objects of the ScheduledEventInformation type being created.

1.4 Prerequisites

• You need to use Advanced Audit Trail with Studio Pro versions starting with 9.12.
• You need to have a subscription to the Advanced Audit Trail solution to store your data in an external data storage that runs in the Mendix Cloud.

1.5 Dependencies

• You need to install Atlas Core and Atlas Web Content in your application.

2 Installation

2.1 Starting a Subscription

Advanced Audit Trail is a premium Mendix product that is subject to a purchase and subscription fee. You can deploy Advanced Audit Trail locally or in a Mendix Free App for free. However, to deploy Advanced Audit Trail on the cloud, you need to start a subscription to get a license token and configure it later. To start a subscription, contact your Customer Success Manager (CSM) or the Mendix Sales organization.

2.2 Installing the Components in Your App

To install the component, click the Contact Us button on the Advanced Audit Trail Marketplace page. Then follow the instructions in the Importing Content from the App Explorer section in Using Marketplace Content to import the Advanced Audit Trail Core module and the Advanced Audit Trail UI module into your app.

3 Configuration

1. Set up your application roles to include the right module roles.

2. Configure the right constant values for the right snapshots.

3. Implement the Before Commit (BCo) and Before Delete (Bde) events. Use the events on the domain model settings (BCo / BDe). For example, the configuration in the image below is for the Before Commit handler, whereas for the Before Delete handler, the value of Is delete should be set to true.

   

   In case you need to follow a compliance that requires you never to delete an object, implement this outside the context of the audit trail module.

   You can create CommitList microflows that commit a list of objects without events, but use Create Snapshot (List) from the Toolbox (the JA_Object_CreateSnapshot_List action). This ensures that the snapshots are committed in a list as well, therefore minimizing the performance impact of the module. When an object is committed without events, this change is not audited unless you explicitly add Create Snapshot (the JA_Object_CreateSnapshot action) or Create Snapshot (List) (the JA_Object_CreateSnapshot_List action) before the commit.

   When your Mendix application includes entities with inheritance, Mendix recommends only applying the event handler on the generalization of this entity. There are cases where it makes sense to apply the event handler on the specialization instead, but applying the event handler to both the generalization and specialization will lead to duplicate snapshots of the same action.
   
   When there are multiple Before Commit (BCo) or Before Delete (Bde) events that may change the object, the order is not guaranteed. See Event Handlers. This means that some changes could theoretically fall outside the context of an audit.

4. Add the open search page microflow AdvancedAuditTrailUI.ACT_SnapshotQuery_CreateAndShowSearch to the navigation.

5. Make sure that the scheduled events are enabled in the cloud portal.

   Due to protected modules, we do not show scheduled events in Studio Pro.

3.1 Configuring Module Roles

• Admin: The admin can query the entire database for the current application and can access the debug pages.

• _AddOn_CanChangeEnvironmentInQuery: This is an additional role for the Admin, which allows the Admin to change the environment in search queries so that they can also search in other applications.

• DisplayOnly: The display-only user can view queries that are prepared in microflows, but they cannot change any of them. This can restrict the user to seeing information they are allowed to see. The role is tested against cross-site scripting (XSS).

  Access from and to the long-term data storage is based on service accounts. This means that once a user can access the Snippet_Settings, they can access all data in the long-term storage, even if it belongs to other applications in the same environment. Any user-based authentication needs to be implemented in the runtime, for example, by using the DisplayOnly module role and the Query Snapshots for object setup.

3.2 Configuring Constants

• Retention settings for the local cached data

  • SnapshotRetentionDays: This is the days that the records should be kept in the local snapshot cache.
  • OnlyDeleteProcessedItems: This indicates whether items should only be deleted if they are sent to the external data storage.
    • If OnlyDeleteProcessedItems is set to True, SnapshotRetentionDays only applies to processed snapshots.

• Snapshots

  • IncludeHashedStrings: This indicates whether to include attributes of type hashed string (for example, password fields) in the snapshots.

    • True: Hashed strings (storing bcrypt or other hashed value) will be included.
    • False: Hashed strings will be excluded and, therefore, not audited.
    Manually-encrypted (for example, using the Encryption module) strings are not the type of hashed string and will not be affected by this setting.

• Integration

  • EnvironmentName: This is the name of the environment within Kibana, which should be unique in your audit data storage, for example, myApp-prod. Do not use any whitespace or tilde (~) for the environment name.

    If two applications use the same name, the audit trail will not be able to distinguish between the two, effectively breaking the audit trail for both applications irreversibly.

  • EnvironmentURL (optional): This URL is used to identify the environment. If left empty, the application runtime URL is used instead.

  • Kafka_Endpoint / Kafka_Username and Kafka_Password: These are the credentials for the Kafka environment for sending the data into the long-term storage.

  • Kibana_Endpoint / Kibana_Username and Kibana_Password: These are the credentials for the Kibana environment for receiving the data from the long-term storage.

3.3 Configuring Scheduled Events

• SE_SendAuditSnapshots: This sends the cached data to the external data storage. This occurs each minute.

• SE_CleanupSnapshotCache: This cleans up the cached data based on the retention settings—OnlyDeleteProcessedItems and SnapshotRetentionDays. This occurs daily at 3:00 AM UTC.

• SE_PeriodicVacuum: This runs a periodic vacuum on a PostgreSQL database. This is not needed on Microsoft SQL. Other database types are not supported. This occurs every 2 hours.

  Enable the scheduled event SE_PeriodicVacuum in the cloud portal for PostgreSQL databases. PostgreSQL databases require a regular VACUUM when the application creates and deletes many objects in order to stay quick and not grow out of disk space. The default Mendix Cloud settings will not always perform the VACUUM when needed. The scheduled event SE_PeriodicVacuum performs the VACUUM regularly. This scheduled event is for PostgreSQL only. For more information, see PostgreSQL documentation on VACUUM and ANALYZE.

3.4 Configuring Advanced Features (Optional)

• Link NAV_AdvancedSettings for accessing debug settings. Typically not needed, the features here are subject to change.
• Link NAV_CachedSnapshot_Overview to access the local cache of snapshot data.
• Check the default values of the NPE Settings object in the domain model, since they will be used to configure the app.
• Enable SE_CleanupAuditSnapshots if you want to use the retention settings and delete objects.

3.5 Adding Additional Information to a Snapshot (Optional)

It is possible to submit additional information for a snapshot (for example, in order to provide a rationale on why the said action has taken place on the object in question). Developers can configure this feature for certain actions (creation, deletion, updating). To use this feature, the developer must use Set additional info for snapshots from the Toolbox (the JA_SetAdditionalInfo action) to set additional information for snapshots.

3.6 Implementing Custom User Logging (Optional)

Use Override User for Snapshots in this Context from the Toolbox (the JA_SetUserForSnapshotsaction action) to override the logged user for a request. For example, the request is a published REST service that runs in a system context, while the user is known.

3.7 Implementing User Name Scrambling (Optional)

Use Configure Username mapping from the Toolbox (the JA_ConfigureUsernameMapping action) to store a username differently in the long-term data storage. This can be used for anonymizing data (for example, due to GDPR).

3.8 Implementing Display Formatters (Optional)

Use the formatter microflows to change how externally-stored values will be displayed inside your Mendix application.

3.9 Getting Microflow Stack Trace (Optional)

Use Get microflow stack trace from the Toolbox (the JA_GetMicroflowTrace action) to create custom logging/entities and identify in what microflow the action was triggered.

3.10 Changing the Module Layout (Optional)

Update the AuditSnapshots_ResponsiveLayout to update the layouts without changing the pages.

3.11 Configuring the License Key

You can deploy Advanced Audit Trail locally or in a Mendix Free App for free. However, to deploy Advanced Audit Trail in the Mendix Cloud, you need to start a subscription to obtain a license key and then configure it.

Before you deploy your app, configure the app Constants in the deployment package in the Developer Portal.

If you have already deployed your app, change the existing Licensekey constant value on the Model Options tab and restart the app.

4 Read More

• Consuming Add-on Modules and Solutions