GenAI Commons

Introduction

The GenAI Commons module combines common GenAI patterns found in a variety of generative AI models on the market. Platform-supported GenAI-connectors use the underlying data structures and their operations. This makes it easier to develop vendor agnostic AI-enhanced apps with Mendix, for example by using one of the connectors or the Conversational UI module.

If two different connectors both adhere to the GenAI Commons module, they can be easily swapped, which reduces dependency on the model providers. In addition, the initial implementation of AI capabilities using the connectors becomes a drag and drop experience, so that developers can quickly get started. The module exposes useful operations which developers can use to build a request to a large language model (LLM), and to handle the response.

Developers who want to connect to another LLM provider or their own service are advised to use the GenAI Commons module as well. This speeds up the development and ensures that common principles are taken into account. Lastly, other developers or consumers of the connector can adapt to it more quickly.

Limitations

The current scope of the module is focused on text and image generation, as well as embeddings and knowledgebase use cases.

Dependencies

The GenAI Commons module requires Mendix Studio Pro version 9.24.2 or above.

You must also install and configure the Community Commons module.

Installation

If you are starting from the Blank GenAI app, or the AI Bot Starter App, the GenAI Commons module is included and does not need to be installed manually.

If you start from a blank app, or have an existing project where you want to include a connector for which the GenAI Commons module is a required module, you must install GenAI Commons manually. First, install the Community commons module, and then follow the instructions in using Marketplace content to import the GenAI Commons module into your app.

Implementation

GenAI Commons is the foundation of chat completion implementations within the OpenAI connector and the Amazon Bedrock connector, but may also be used to build other GenAI service implementations on top of it by reusing the provided domain model and exposed microflows.

Although GenAI Commons technically defines additional capabilities typically found in chat completion APIs, such as image processing (vision) and tools (function calling), it depends on the connector module of choice for whether these are actually implemented and supported by the LLM. To learn which additional capabilities a connector supports and for which models these can be used, refer to the documentation of that connector.

The GenAI Commons module is protected, which means that it cannot be changed and the logic of the microflows is not visible. For information about what each exposed operation does, see Microflows, or refer to the documentation inside the module.

Technical Reference

The technical purpose of GenAI Commons module is to define a common domain model for generative AI use cases in Mendix applications. To help you work with the GenAI Commons module, the following sections list the available entities, enumerations, and microflows that you can use in your application.

Domain Model

The domain model in Mendix is a data model that describes the information in your application domain in an abstract way. For more general information, see the Domain Model documentation. To learn about where the entities from the domain model are used and relevant during implementation, see the Microflows section below.

Connection

The Connection entity contains specifications to interact with an AI provider.

Request

Request is an input object for the chat completions operations defined in the platform-supported GenAI-connectors and contains all content-related input needed for an LLM to generate a response for the given chat conversation.

Message

A message that is part of the request or the response. Each instance contains data (text, file collection) that needs to be taken into account by the model when processing the completion request.

FileCollection

This is an optional collection of files that is part of a Message. It is used for patterns like vision, where image files are sent along with the user message for the model to process. It functions as a wrapper entity for files and has no attributes.

FileContent

This is a file in a collection of files that belongs to a message. Each instance represents a single file. Currently only files of the type image and document are supported.

ToolCollection

This is an optional collection of tools to be sent along with the Request. Using tool call capabilities (also known as function calling) might not be supported by certain AI providers or models. This entity functions as a wrapper entity for tools and has no attributes.

Tool

A tool in the tool collection. This is sent along with the request in order to expose a list of available tools. In the response, the model can suggest to call a certain tool (or multiple tools in parallel) in order to retrieve additional data or perform certain actions.

Function

A tool of the type function. This is a specialization of Tool, and represents a microflow in the same Mendix application. The return value of this microflow when executed as function is sent to the model in a next iteration and hence must be of type String.

StopSequence

For many models, StopSequence can be used to pass a list of character sequences (for example a word) along with the request. The model will stop generating content when a word of that list would occur next.

Response

The response returned by the model contains usage metrics as well as a response message.

ToolCall

A tool call object may be generated by the model in certain scenarios, such as a function call pattern. This entity is only applicable for messages with role assistant.

Reference

An optional reference for a response message.

Citation

An optional citation. This entity can be used to visualize the link between a part of the generated text and the actual text in the source on which the generated text was based.

ChunkCollection

This entity represents a collection of chunks. It is a wrapper entity for Chunk objects or specialization(s) to pass it to operations that execute embedding calculations or knowledge base interaction.

Chunk

A piece of information (InputText) and the corresponding embeddings vector retrieved from an Embeddings API.

KnowledgeBaseChunk

This entity represents a discrete piece of knowledge that can be used in embed and store operations. It is a specialization of Chunk.

MetadataCollection

An optional collection of metadata. This is a wrapper entity for one or more Metadata objects for a KnowledgeBaseChunk.

Metadata

This entity represents additional information that is to be stored with the KnowledgeBaseChunk in the knowledge base. It can be used for custom filtering during retrieval.

EmbeddingsOptions

An optional input object for the embeddings operations to set optional request attributes.

EmbeddingsResponse

The response returned by the model contains token usage metrics. Not all connectors or models might support token usage metrics.

ImageOptions

An optional input object for the image generations operations to set optional request attributes.

Enumerations

ENUM_MessageRole

ENUM_MessageRole provides a list of message author roles.

ENUM_MessageType

ENUM_MessageType provides a list of ways of interpreting a message object.

ENUM_ContentType

ENUM_ContentType provides a list of possible file content types, which describe how the file data is encoded in the FileContent attribute on the FileContent object that is part with the Message.

ENUM_FileType

ENUM_FileType provides a list of file types. Currently only image and document is a supported file type. Not all file types might be supported by all AI providers or models.

ENUM_ToolChoice

ENUM_ToolChoice provides a list of ways to control which (if any) tool is called by the model. Not all tool choices might be supported by all AI providers or models.

ENUM_SourceType

ENUM_SourceType provides a list of source types, which describe how the pointer to the Source attribute on the Reference object should be interpreted to get the source location. Currently, only Url is supported.

ENUM_ImageGenerationType

ENUM_ImageGenerationType describes how the image generations operation is to be used. Currently only text to image is supported.

Microflows activities

Use the exposed microflows and Java Actions to map the required information for GenAI operations from your custom app implementation to the GenAI model and vice versa. Two sets of operations are provided: one for text and files, plus a second one for embeddings and knowledge bases.

Text and Files: Build request

The following microflows help you construct the input request structures for the operations for text and files defined in GenAI Commons.

Chat: Create Request

This microflow can be used to create a request for a chat completion operation. This is the request object that contains the top-level functional input for the language model to generate a response.

Input Parameters

Return Value

Chat: Add Message to Request

This microflow can be used to add a new Message to the Request object. A message represents the conversation text content and optionally has a collection of files attached that need to be taken into account when generating the response (such as images for vision). Make sure to add messages chronologically so that the most recent message is added last.

Input Parameters

Return Value

This microflow does not have a return value.

Chat: Add Stop Sequence

This microflow can be used to add an optional StopSequence to the request. It can be used after the request has been created. If available for the connector and model of choice, stop sequences let models know when to stop generating text.

Input Parameters

Return Value

This microflow does not have a return value.

Files: Initialize Collection with File

In order to include files within a message, you must provide them in the form of a file collection. This helper microflow creates the file collection and adds the first file. The File Collection is an optional part of a Message object.

Input Parameters

Return Value

Files: Add File to Collection

Use this microflow to add a file to an existing FileCollection. The File Collection is an optional part of a Message.

Input Parameters

Return Value

This microflow does not have a return value.

Image: Create ImageOptions

This microflow creates new ImageOptions.

Input Parameters

Return Value

Tools: Add Function to Request

Adds a new Function to a ToolCollection that is part of a Request. Use this microflow when you have microflows in your application that may be called to retrieve the required information as part of a GenAI interaction. If you want the model to be aware of these microflows, you can use this operation to add them as functions to the request. If supported by the LLM connector, the chat completion operation calls the right functions based on the LLM response and continues the process until the assistant’s final response is returned.

Input Parameters

Return Value

Tools: Set Tool Choice

Use this microflow to control how the model should determine which function to leverage (typically to gather additional information). The microflow sets the ToolChoice within a Request. This controls which (if any) function is called by the model. If the ENUM_ToolChoice equals tool, the Tool input is required which will become the tool choice. This will force the model to call that particular tool.

Input Parameters

Return Value

This microflow does not have a return value.

Text and Files: Handle Response

The following microflows handle the response processing.

Get Model Response Text

This microflow can be used to get the content from the latest assistant message over association Response_Message. Use this microflow to get the response text from the latest assistant response message. In many cases, this is the main value needed for further logic after the operation or is displayed to the end user.

Input Parameters

Return Value

Get References

Use this microflow to get the list of references that may be included in the model response. These can be used to display source information, content, and citations on which the model response text was based according to the language model. References are only available if they were specifically requested from the LLM and mapped from the LLM response into the GenAI Commons domain model.

Input Parameters

Return Value

Get Generated Image (Single)

This operation processes a response that was created by an image generations operation. A return entity can be specified using ResponseImageEntity (needs to be of type System.Image or its specialization). An image of that type will be created and returned.

Input Parameters

Return Value

Get Generated Images (List)

This operation processes a response that was created by an image generations operation. A return entity can be specified using ResponseImageEntity (needs to be of type System.Image or its specialization). A list of images of that type will be created and returned.

Input Parameters

Return Value

Text and Files: Chat Completions Interface

The OpenAI connector and the Amazon Bedrock connector both have two chat completions and image generations operations implemented that share the same interface, meaning that they expect the same entities as input and as output. This has the advantage that these operations can be exchanged very easily without much additional development effort.

We recommend that you adapt to the same interface when developing custom chat completions or image generations operations, such as integration with different AI providers. The generic interfaces are described below. For more detailed information, refer to the documentation of the connector that you want to use, since it may expect specializations of the generic GenAI common entities as an input.

Chat Completions (Without History)

The Chat Completions (without history) operation interface supports scenarios where there is no need to send a list of (historic) messages comprising the conversation so far as part of the request.

Input Parameters

Return Value

Chat Completions (With History)

The Chat Completions (with history) operation interface supports more complex use cases where a list of (historical) messages (for example, comprising the conversation or context so far) is sent as part of the request to the LLM.

Input Parameters

Return Value

Image Generations

The Image Generations operation interface supports the generation of images based on a UserPrompt passed as string. The returned Response contains a FileContent via FileCollection and Message. See microflows in the Connector Building folder to construct the output.

Input Parameters

Return Value

Knowledge Bases and Embeddings

The following microflows and Java actions help you construct the input structures and handle the response object for the operations for knowledge bases and embeddings as defined in GenAI Commons.

Chunks: Initialize Chunkcollection

This microflow creates a new ChunkCollection and returns it.

Input Parameters

This microflow has no input parameters.

Return Value

Chunk: Add Chunk to ChunkCollection

This microflow adds a new Chunk to the ChunkCollection.

Input Parameters

Return Value

Chunk: Add KnowledgeBaseChunk to ChunkCollection

This Java action adds a new KnowledgeBaseChunk to the ChunkCollection to create the input for embeddings or knowledge base operations. Optionally, a MetadataCollection can be added for more advanced filtering. Use Initialize MetadataCollection with Metadata to instantiate a MetadataCollection first, if needed.

Input parameters

Return Value

Embeddings: Create EmbeddingsOptions

This microflow creates new EmbeddingsOptions.

Input Parameters

Return Value

Embeddings: Get First Vector from Response

This microflow gets the first embedding vector from the response of an embeddings operation.

Input Parameters

Return Value

Knowledge Base: Initialize MetadataCollection with Metadata

This microflow creates a new MetadataCollection and adds a new Metadata. The MetadataCollection will be returned. To add additional Metadata, use Add Metadata to MetadataCollection.

Input Parameters

Return Value

Knowledge Base: Add Metadata to MetadataCollection

This microflow adds a new Metadata object to a given MetadataCollection. Use Initialize MetadataCollection with Metadata to instantiate a MetadataCollection first, if needed.

Input Parameters

Return Value

This microflow does not have a return value.

Knowledge Bases and Embeddings: Embeddings Interface

To make use of embeddings in a Mendix app, GenAI Commons defines interfaces for embedding operations that connectors can adhere to. We recommend that you adapt to the same interface when developing custom embedding operations, such as integration with different AI providers. The generic interfaces are described below. For more detailed information, refer to the documentation of the connector that you want to use, since it may expect specializations of the generic GenAI common entities as an input.

Embeddings (String)

The Embeddings (String) operation interface allows the invocation of the embeddings API with a String input and returns an EmbeddingsResponse object with token usage statistics, if applicable. The EmbeddingsResponse_GetFirstVector microflow from GenAI Commons can be used to retrieve the corresponding embedding vector in a String representation. This operation supports scenarios where the vector embedding of a single string must be generated, e.g. to perform a nearest neighbor search across an existing knowledge base.

Input Parameters

Return Value

Embeddings (ChunkCollection)

The Embeddings (ChunkCollection) operation interface allows the invocation of an embeddings API with a ChunkCollection and returns an EmbeddingsResponse object with token usage statistics, if applicable. The response object is associated with the original ChunkCollection used as an input, and the Chunk (or KnowledgeBaseChunk) objects will be updated with their corresponding embedding vector retrieved from the Embeddings API within this microflow.

Input Parameters

Return Value