OpenAI
1 Introduction
The OpenAI Connector allows you to integrate generative AI into your Mendix app and is compatible with OpenAI’s platform as well as Azure’s OpenAI service.
The current scope covers text generation use cases based on the OpenAI Chat Completions API, image generation use cases based on the Image Generations API, and embeddings use cases based on the Embeddings API.
Image generations with the DALL-E models is currently supported only by OpenAI, while Azure OpenAI only supports it in preview mode. For all other operations, Mendix provides dual platform support for OpenAI as well as Azure OpenAI.
1.1 Typical Use Cases
1.1.1 Text Generation
- Develop interactive AI chatbots and virtual assistants that can carry out conversations in a natural and engaging manner.
- Use OpenAI’s large language models for text comprehension and analysis use cases such as summarization, synthesis, and answering questions about large amounts of text.
- Fine-tune the OpenAI models on a specific task or domain, by training it on custom data, to improve its performance.
- Integrate more easily with OpenAI’s platform which, by providing text generation models, allows you to build applications with the following features:
- Draft documents
- Write computer code
- Answer questions about a knowledge base
- Analyze texts
- Give software a natural language interface
- Tutor in a range of subjects
- Translate languages
- Simulate characters for games
OpenAI provides market-leading large language model capabilities with GPT-4:
- Advanced reasoning: Follow complex instructions in natural language and solve difficult problems with accuracy.
- Creativity: Generate, edit, and iterate with users on creative and technical writing tasks, such as composing songs, writing screenplays, or learning a user’s writing style.
- Longer context: GPT-4 is capable of handling over 25,000 words of text, allowing for use cases like long form content creation, extended conversations, and document search and analysis.
1.1.2 Image Generation
Generate one or more completely new, original images and art from a text description. Powered by the OpenAI DALL-E models, the connector enables developers to generate these images by combining concepts, attributes, and styles.
1.1.3 Embeddings
Convert strings into vector embeddings for various purposes based on the relatedness of texts. Embeddings are commonly used for:
- Search
- Clustering
- Recommendations
- Anomaly detection
- Diversity measurement
- Classification
Combine embeddings with text generation capabilities and leverage specific sources of information to create a smart chat functionality tailored to your own knowledge base.
1.2 Features
Mendix provides dual platform support for both OpenAI and Azure OpenAI.
With the current version, Mendix supports the Chat Completions API for text generation, the Image Generations API for images, and the Embeddings API for vector embeddings.
1.3 Limitations
The current scope of the connector is focused on text and image generation use cases, as well as embeddings. We try to release early and often, so keep your eyes open for new releases!
1.4 Prerequisites
You should have signed up for an OpenAI account, or have access to deployments at Azure OpenAI.
1.5 Dependencies
- Mendix Studio Pro version 9.24.0 or higher
- Encryption module
- Community commons module
2 Installation
Follow the instructions in Using Marketplace Content to import the OpenAI Connector into your app.
3 Configuration
After you install the OpenAI Connector, you can find it in the App Explorer, in the Marketplace modules section. The connector provides a domain model and several activities that you can use to connect your app to OpenAI. Each activity can be implemented by using it in a microflow. To ensure that your app can connect to OpenAI, you must also configure the Encryption module.
3.1 General Configuration
- Add the module role OpenAIConnector.Administrator to your Administrator user role in the security settings of your app.
- Add the Configuration_Overview page (USE_ME > Configuration) to your navigation or add the Snippet_Configurations to a page that is already part of your navigation.
- Continue to set up your OpenAI configuration at runtime. Based on the type of your configuration, continue with one of the following sections:
3.1.1 OpenAI Configuration
The following inputs are required for the OpenAI configuration:
Parameter | Value |
---|---|
DisplayName | This is the name identifier of a configuration, e.g. MyConfiguration. |
API type | Select OpenAI .For more information, see the ENUM_ApiType section. |
Endpoint | This is the API Endpoint, e.g. https://api.openai.com/v1 |
API key | This is the access token to authorize your API call. To get an API, follow these steps:
|
3.1.2 Azure OpenAI Configuration
The following inputs are required for the Azure OpenAI configuration:
Parameter | Value |
---|---|
DisplayName | This is the name identifier of a configuration, e.g. MyConfiguration. |
API type | Select AzureOpenAI .For more information, see the ENUM_ApiType section. |
Endpoint | This is the API Endpoint, e.g. https://your-resource-name.openai.azure.com/openai/deployments/ .For more information about how to obtain your-resource-name , see the Obtaining Azure OpenAI Resource Name section below. |
DeploymentName | This is the deployment name you chose when you deployed the model. Deployments provide endpoints to the Azure OpenAI base models, or your fine-tuned models. To check the deployment name, go to Azure OpenAI and check the deployment name under Deployments. |
API version | This is the API version to use for this operation. This follows the yyyy-MM-dd format. See Azure OpenAI documentation for supported versions. |
API key | This is the access token to authorize your API call. |
Key type | This is the type of token that is entered in the API key field. For Azure OpenAI, two types of keys are currently supported: Microsoft Entra token and API key . For more information about how to generate a Microsoft Entra access token, see How to Configure Azure OpenAI Service with Managed Identities. Alternatively, if your organization allows it, you could use the Azure api-key authentication mechanism. For more information about how to obtain an API key , see the Obtaining Azure OpenAI API keys section below. For more information, see the ENUM_KeyType section. |
3.1.2.1 Obtaining Azure OpenAI Resource Name
- Go to the Azure OpenAI portal and sign in.
- On the upper-right corner, click Settings ().
- Go to the Resource tab.
- Go to Current resource and click JSON view.
- Use the value of the name field as your resource name in the endpoint URL.
3.1.2.2 Obtaining Azure OpenAI API keys
- Go to the Azure OpenAI portal and sign in.
- On the upper-right corner, click Settings ().
- Go to the Resource tab.
- Go to Current resource and click JSON view.
- Use the value of the key1 or key2 field as your API key while setting up the configuration. Note that these keys might not be available depending on your organization’s security settings.
3.2 Chat Completions Configuration
After following the general setup above, you are all set to use the microflows in the USE_ME > Operations > ChatCompletions folder in your logic. Currently, three microflows for chat completions are exposed as microflow actions under the OpenAI Connector category in the Toolbox in Mendix Studio Pro.
These microflows expect a Configuration entity, as well as the desired AI model that should be used for generating responses.
- For the OpenAI API configuration, the desired model must be specified for every call.
- For the Azure OpenAI configuration, the model is already determined by the deployment in the Azure OpenAI portal. Any model explicitly specified will be ignored and hence can be left empty.
In the context of chat completions, system prompts and user prompts are two key components that help guide the language model in generating relevant and contextually appropriate responses. For more information on prompt engineering, see the Read More section. It varies per exposed microflow activity which prompts are required and how these must be passed, as described in the following sections. For more information, see the ENUM_Role section.
For more inspiration or guidance on how to use the above-mentioned microflows in your logic, Mendix highly recommends downloading our showcase app from the Marketplace that displays a variety of examples.
3.2.1 Chat Completions (without history)
The microflow activity Chat Completions (without history)
supports scenarios where there is no need to send a list of (historic) messages comprising the conversation so far as part of the request. The system prompt and user prompt are available as string input parameters. Depending on the use case, both or only one can be used. For technical details, see the Technical reference section.
Functionally, the prompt strings can be written in a specific way and can be tailored to get the desired result and behavior. For more information on prompt engineering, see the Read More section.
3.2.2 Chat Completions (with history)
The microflow activity Chat completions with history
supports more complex use cases where a list of (historical) messages (e.g. comprising the conversation or context so far) is sent as part of the request to the language model. Two accompanying microflows are available to construct the input for the microflow:
ChatCompletionsSession_Create
is used to create the session wrapper that must be passed as input parameter.ChatCompletionsSession_AddMessage
is used to attach the historical messages to theChatCompletionsSession
.
The content of such a message corresponds to a system, assistant, or user prompt. In the case of multiple historical messages the order is relevant.
For technical details, see the Technical reference section.
3.2.3 Chat Completions (advanced)
The microflow activity Chat Completions (advanced)
can be used in cases where the above-mentioned microflows do not provide enough support or flexibility. The interface of this operation resembles the API interface. The construction of the request and handling of the response must be implemented in a custom way. Three accompanying microflows are available to construct the input for the microflow:
ChatCompletionsRequest_Create
is used to create the request object.ChatCompletionsMessages_Create
is used to create the wrapper object for theChatCompletionsMessageRequest
objects.ChatCompletionsMessageRequest_Create
is used to create the message objects.
The construction of the request and handling of the response must be implemented in a custom way.
For technical details, see the Technical reference section.
3.3 Image Generations Configuration
In order to implement image generations into your Mendix application, you can use the microflows in the USE_ME > Operations > ImageGenerations folder. Currently, two microflows for image generations are exposed as microflow actions under the OpenAI Connector category in the Toolbox in Mendix Studio Pro.
These microflows, similar to the Chat Completions case, expect a Configuration entity, as well as the desired AI model that should be used for generating image responses in the case of OpenAI configurations. In this case the field is optional, as OpenAI assumes a default value dall-e-2
.
For more inspiration or guidance on how to use the below-mentioned microflows in your logic, Mendix highly recommends downloading our showcase app from the Marketplace that displays a variety of examples.
3.3.1 Image Generations (single image)
The microflow activity Image Generations (single image)
supports scenarios where a single image must be generated based on the provided prompt. In order to implement this operation, you must create a specialization of the GeneratedImage entity. For every implementation of this microflow, an instance of this specialization has to be created first and must be passed into the OutputImage
parameter of the microflow. If the call is successful, the image generated by the model will be stored into that object.
For technical details, see the Technical reference section.
3.3.2 Image Generations (advanced)
The microflow activity Image Generations (advanced)
can be used in cases where the above-mentioned microflows do not provide enough support or flexibility. The interface of this operation resembles the API interface. The construction of the request and handling of the response must be implemented in a custom way. One accompanying microflow is available to construct the input for the microflow:
ImageGenerationsRequest_Create
is used to create the request object.
The construction of the request and handling of the response must be implemented in a custom way.
For technical details, see the Technical reference section.
3.4 Embeddings Configuration
In order to implement embeddings into your Mendix application, you can use the microflows in the USE_ME > Operations > Embeddings folder. Currently, three microflows for embeddings are exposed as microflow actions under the OpenAI Connector category in the Toolbox in Mendix Studio Pro.
These microflows expect a Configuration entity, as well as the desired AI model that should be used for generating responses.
- For the OpenAI API configuration, the desired model must be specified for every call.
- For the Azure OpenAI configuration, the model is already determined by the deployment in the Azure OpenAI portal. Any model explicitly specified will be ignored and hence can be left empty.
3.4.1 Embeddings (single input)
The microflow activity Embeddings (single input)
supports scenarios where the vector embedding of a single string must be generated. This input string can be passed directly as the Input
parameter of this microflow. Note that the parameter EncodingFormat
is optional; the current version of this operation only supports the float representation of the resulting vector.
For technical details, see the Technical reference section.
3.4.2 Embeddings (list input)
The microflow activity Embeddings (list input)
supports the more complex scenario where a list of strings must be vectorized in a single API call, e.g. converting a batch of text strings (chunks) from a private knowledge base into embeddings. Instead of calling the API for each string, executing a single call for a list of strings can significantly reduce HTTP overhead. The embedding vectors returned after a successful API call will be stored as EmbeddingVector
attribute in the same DataChunk
entity. Thus, the microflow does not return an Object or List, but only a Success
Boolean. Two accompanying microflows are available to help construct the input for the main microflow:
DataBatch_Create
is used to create the wrapper object for the list ofDataChunk
objects that must be passed as input parameter.DataChunk_Create
can be used repetitively to attach a chunk of text (as a string) to theDataBatch
entity.
For technical details, see the Technical reference section.
3.4.3 Embeddings (advanced)
The microflow activity Embeddings (advanced)
can be used in cases where the above-mentioned microflows do not provide enough support or flexibility. The interface of this operation resembles the API interface. Two accompanying microflows are available to help construct the input for the main microflow:
EmbeddingsRequest_Create
is used to create the request object.EmbeddingsInput_Create
is used to create the input object.
The construction of the request and handling of the response must be implemented in a custom way.
For technical details, see the Technical reference section.
4 Technical Reference
To help you work with the OpenAI Connector, the following sections list the available entities, enumerations, and activities that you can use in your application.
4.1 Domain Model
The domain model in Mendix is a data model that describes the information in your application domain in an abstract way. For more general information, see Domain model. To learn about where the entities from the domain model are used and relevant during implementation, see the Activities section below.
4.1.1 Configuration
4.1.1.1 Configuration
This entity is used to store the API credentials and endpoints in the configuration for OpenAI or Azure OpenAI .
Attribute | Description |
---|---|
DisplayName |
This is the name identifier of a configuration. |
ApiType |
The value can be OpenAI or AzureOpenAI .For more information, see the ENUM_ApiType section. |
Endpoint |
This is the API Endpoint, e.g. https://api.openai.com/v1 for OpenAI, or https://your-resource-name.openai.azure.com/openai/deployments/ for Azure OpenAI. |
DeploymentName |
This is the deployment name you chose when you deployed the model. This is only relevant for configurations of ApiType AzureOpenAI. Deployments provide endpoints to the Azure OpenAI base models, or your fine-tuned models.To check the deployment name, follow these steps:
|
ApiVersion |
This the API version used for this operation. This follows the YYYY-MM-DD format. Only relevant for configurations of ApiType AzureOpenAI. |
ApiKey |
This is the access token to authorize your API call. For details, see the OpenAI configuration and Azure OpenAI configuration sections. |
KeyType |
This is the type of token entered in the ApiKey field. This is only relevant for configurations of ApiType AzureOpenAI.For more information, see the ENUM_ApiType section. |
4.1.1.2 ApiKey
This entity is only used for editing the ApiKey
to be stored in the Configuration entity.
Attribute | Description |
---|---|
ApiKey |
This is the access token to authorize your API call. |
4.1.1.3 ConfigurationTest
This entity is only used to send a simple chat completions request to test an existing configuration.
Attribute | Description |
---|---|
UserPrompt |
This is the user message sent to the chat completions API. |
AssistantResponse |
This is the assistant response returned by the chat completions API. |
ChatCompletionsModel |
This is the model used for the API call. |
4.1.2 Generalizations
4.1.2.1 AbstractUsage
This entity contains usage statistics for an API call. Do not use this entity directly. Instead, use one of its specializations.
Attribute | Description |
---|---|
Prompt_tokens |
This is the number of tokens in the prompt. |
Total_tokens |
This is the total number of tokens used in the request. |
For more information on how to manage tokens for text generation, see Managing tokens.
4.1.2.2 AbstractChatCompletionsMessage
This is the abstract entity for ChatCompletionsMessage
. Do not use this entity directly. Instead, use one of its specializations.
Attribute | Description |
---|---|
Content |
This is the content of a message. |
Role |
This is the role of the message author. For more information, see the ENUM_Role section. |
4.1.3 Chat Completions
4.1.3.1 ChatCompletionsRequest
A chat completions request that creates a model response for the given chat conversation.
Attribute | Description |
---|---|
Model |
This is required for requests to OpenAI. Model is NOT considered for request to Azure OpenAI, because the model is determined by the deployment. For more information, see the compatible models in the OpenAI documentation. |
Frequency_penalty |
The value should be a decimal between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model’s likelihood of repeating the same line verbatim. This attribute is optional. The default value is 0.0. |
Max_tokens |
This is the maximum number of tokens to generate in the chat completion. The total length of input tokens and generated tokens is limited by the model’s context length. This attribute is optional. |
Temperature |
This is the sampling temperature. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. This attribute is optional. The value should be a decimal between 0.0 and 2.0. The default value is 1.0. |
Top_p |
This is an alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with Top_p probability mass. 0.1 means only the tokens comprising the top 10% probability mass are considered. Mendix generally recommends altering Top_p or Temperature but not both. This attribute is optional. The value should be a decimal between 0.0 and 1.0. The default value is 1.0. |
N |
This is the number of chat completions choices to generate for each input message. You will be charged based on the number of generated tokens across all choices. Keep N as 1 to minimize costs. This attribute is optional. The default value is 1. |
4.1.3.2 ResponseFormat
This specifies the format that the chat completions model must output.
Attribute | Description |
---|---|
_Type |
This describes the format that the chat completions model must output. For more information, see the ENUM_ResponseFormat_Chat section. |
4.1.3.3 ChatCompletionsMessages
This is a wrapper for a list of messages comprising the conversation so far.
4.1.3.4 ChatCompletionsMessageRequest
This is a specialization of the AbstractChatCompletionsMessage entity. Each instance contains a text that needs to be taken into account by the model when processing the completion request.
4.1.3.5 ChatCompletionsResponse
This represents a chat completion response returned by the model, based on the provided input.
Attribute | Description |
---|---|
_id |
This is a unique identifier for the chat completion. |
_object |
This is the object type, which is always chat.completion. |
Created |
This is the Unix timestamp (in seconds) of when the chat completion was created. |
Model |
This is the model that was used for the chat completion. |
System_fingerprint |
This fingerprint represents the backend configuration that the model runs with. The value can be used in conjunction with the seed request parameter to understand when backend changes have been made that might impact determinism. |
4.1.3.6 Choice
This is a list of chat completion choices which are part of the response. There can be more than one choice if N
in the request is greater than 1, meaning that there was an explicit request for multiple alternative response texts. Each is used as a wrapper entity for the actual message content.
Attribute | Description |
---|---|
Index |
This is the index of the choice in the list of choices. |
Finish_reason |
This is the reason the model stopped generating tokens. This will be stop if the model hit a natural stop point or a provided stop sequence, length if the maximum number of tokens specified in the request was reached, content_filter if content was omitted due to a flag from our content filters, or tool_calls if the model called a tool. |
4.1.3.7 ChatCompletionsMessageResponse
This is a specialization of the AbstractChatCompletionsMessage entity. It contains the response text (assistant prompt).
4.1.3.8 ChatCompletionsUsage
This is a specialization of the AbstractUsage. It contains the statistics for the completion request with an additional attribute:
Attribute | Description |
---|---|
Completion_tokens |
This is the number of tokens in the generated completion. |
4.1.3.9 ChatCompletionsSession
This entity functions as a wrapper object for a chat completions session. It is associated with a list of (historical) messages comprising the conversation so far that can be mapped to the chat completions request.
4.1.3.10 ChatCompletionsSessionMessage
This is a specialization of the AbstractChatCompletionsMessage entity.
4.1.4 Image Generations
4.1.4.1 ImageGenerationsRequest
This is an image generations request that creates a model response including generated image (or images) for the given prompt.
Attribute | Description |
---|---|
Prompt |
This is the prompt that is used by the model to generate the image (or images) . |
Model |
The model to use for image generation. This is an optional field for OpenAI. Its default value is dall-e-2 . For more information, see the compatible models in the OpenAI documentation. |
N |
This is the number of images to generate. The value must be between 1 and 10. For dall-e-3 , only n=1 is supported. This attribute is optional. |
Quality |
This is the requested quality of the generated images. This attribute is optional and only supported for dall-e-3 . It defaults to standard .For more information, see the ENUM_Quality section. |
ResponseFormat |
This is a parameter used to specify the technical format of the returned generated images by the API. This attribute is optional. The default value is url . For more information see the ENUM_ResponseFormat_Image section. |
Size |
This is the requested size of the generated images. This attribute is optional. Its default value is 1024x1024 .For more information see the ENUM_Size section. |
Style |
This is the style of the generated images. This attribute is optional. Its default value is vivid .For more information, see the ENUM_Style section. |
User |
This is a unique identifier representing your end-user, which can help OpenAI to monitor and detect abuse. This attribute is optional. |
4.1.4.2 ImageGenerationsResponse
This represents an image generations response returned by the model, based on the provided input.
Attribute | Description |
---|---|
Created |
This is the Unix timestamp (in seconds) of when the image generation was created. |
4.1.4.3 Data
This is a wrapper for a list of images that are part of the response.
4.1.4.4 Image
This represents the URL or the content of an image generated by the API.
Attribute | Description |
---|---|
Url |
This is the URL of the generated image that can be used to fetch the image data if the responseFormat is url .
URLs typically expire after a certain time.
|
B64Json |
This is the base64-encoded string representation of the generated image that can be used to process the image data if the responseFormat is b64_json . |
RevisedPrompt |
This is the prompt that was used to generate the image. It is only populated if there was any revision to the prompt. |
4.1.4.5 GeneratedImage
This is an entity that is used to map the image data from the API response onto a Mendix image entity so that it can be used as such in the application.
Attribute | Description |
---|---|
RevisedPrompt |
This is the prompt that was used to generate the image. It is only populated if there was any revision to the prompt. |
4.1.5 Embeddings
4.1.5.1 EmbeddingsRequest
This is an embeddings request that generates a model response including a vector embedding per given input string text.
Attribute | Description |
---|---|
Model |
This is the model used for generating embeddings. This is a mandatory field for OpenAI. For more information, see the compatible models in the OpenAI documentation. |
Encoding_format |
This is the format in which the embeddings are returned. The connector currently only supports float and not base64. For more information see the ENUM_EncodingFormat_Embeddings section. |
User |
This is a unique identifier representing your end-user, which can help OpenAI to monitor and detect abuse. This attribute is optional. |
4.1.5.2 EmbeddingsInput
This is an entity that is used to contain a string input text for the embeddings model.
Attribute | Description |
---|---|
Input |
This is the string input for a text chunk for which the embedding vector needs to be generated. |
4.1.5.3 EmbeddingsResponse
This entity represents an embeddings response returned by the model, based on the provided input.
Attribute | Description |
---|---|
_object |
This is the object type, which is always list . |
Model |
This is the model that has been used for generating the embeddings. |
4.1.5.4 EmbeddingsUsage
This is a specialization of the AbstractUsage entity. It represents usage statistics for the embeddings request that was processed.
4.1.5.5 EmbeddingVector
This is the vector that represents the embedding for the text input that was given in the request. There will be an instance of this entity for every input text string provided.
Attribute | Description |
---|---|
_object |
This is the object type, which is always embedding . |
Index |
This is the index of the embedding in the list of embeddings. This can be used for ordering. |
4.1.5.6 EmbeddingValue
This entity represents an element in the list of floats in the embedding vector returned by the API. It is a separate entity for mapping purposes and is only relevant for the encoding format option float
. The length of the vector depends on the model as listed in the documentation of OpenAI.
Attribute | Description |
---|---|
Value |
This is a decimal in the embedding vector. |
4.1.5.7 DataBatch
This entity functions as a wrapper object for the list input operation for embeddings. It is associated with a list of input objects of entity DataChunk that contain the string texts for which the embedding vectors must be generated.
4.1.5.8 DataChunk
This entity represents a text string, usually a part of a larger base text or discrete piece of text in a data set. It is designed to contain the input string and the corresponding embedding vector retrieved from the Embeddings API.
Attribute | Description |
---|---|
Input |
This is the input text to embed will be mapped to the EmbeddingsInput entity as part of the request. |
EmbeddingVector |
This is the string representation of the embedding vector of the input string. |
Index |
This is used for mapping the EmbeddingVector from the response onto the correct DataChunk in the list. |
4.2 Enumerations
An enumeration is a predefined list of values that can be used as an attribute type. For more information about enumerations in general, see Enumerations.
4.2.1 General
4.2.1.1 ENUM_ApiType
This enumeration provides a list of supported API types.
Name | Caption |
---|---|
AzureOpenAI |
Azure OpenAI |
OpenAI |
OpenAI |
4.2.1.2 ENUM_KeyType
This enumeration provides a list of key types that can be used during the connection to the APIs of Azure OpenAI.
Name | Caption |
---|---|
Bearer_Token |
Microsoft Entra token |
API_key |
API key |
4.2.2 Chat Completions
4.2.2.1 ENUM_Role
This enumeration provides a list of message author roles.
Name | Caption | Description |
---|---|---|
assistant |
Assistant | An assistant message was generated by the model as a response to a user message. |
system |
System | A system message can be used to specify the assistant persona or give the model more guidance and context. This is typically specified by the developer to steer the model response. |
user |
User | A user message is the input from a user. |
4.2.2.2 ENUM_ResponseFormat_Chat
This enumeration provides a list of supported response types for chat completions. Currently chat completions can be returned in normal text format (supported for all chat completions models available in the connector), as well as in JSON mode for specific models.
Name | Caption |
---|---|
json_object |
JSONObject |
text |
Text |
4.2.2.3 ENUM_Quality
This enumeration provides a list of quality levels for the images that are generated.
Name | Caption |
---|---|
standard |
Standard |
hd |
HD |
4.2.3 Image Generations
4.2.3.1 ENUM_ResponseFormat_Image
This enumeration provides a list of supported response types for generated images. Currently, images can be returned either as a URL to a PNG file, or a base64 encoded string representation of the image directly.
Name | Caption |
---|---|
url |
URL |
b64_json |
Base64-JSON |
4.2.3.2 ENUM_Size
This enumeration provides a list of supported pixel dimensions for the generated images. It depends on the model which options are supported.
Name | Caption |
---|---|
_256x256 |
256x256 |
_512x512 |
512x512 |
_1024x1024 |
1024x1024 |
_1204x1792 |
1024x1792 |
_1792x1024 |
1792x1024 |
4.2.3.3 ENUM_Style
This enumeration provides a list of supported visual styles for the generated images. It depends on the model whether this field is supported.
Name | Caption |
---|---|
vivid |
Vivid |
natural |
Natural |
4.2.4 Embeddings
4.2.4.1 ENUM_EncodingFormat_Embeddings
This enumeration provides a list of supported encoding formats for embeddings returned by the API. The connector operations currently only support the floating point representation of embedding vectors and not base64. Therefore, only one value float
exits.
Name | Caption |
---|---|
_float |
float |
4.3 Activities
Activities define the actions that are executed in a microflow or a nanoflow.
4.3.1 Chat Completions
The chat completions API from OpenAI accepts a complex JSON structure that consists of a number of parameters plus one or more messages as input and generates a model-generated message structure as output. While the chat structure is designed for facilitating multi-turn conversations (with history), it is equally valuable for single-turn tasks that do not involve any prior conversation (without history). The exposed microflows in this connector are built to abstract away the complex message structure and are meant to facilitate easier implementation in certain use cases.
4.3.1.1 Chat Completions (Without History)
Use the microflow ChatCompletions_Execute_WithoutHistory
to execute a simple chat completions API call with string input and output not considering a previous conversation. See ENUM_Role for the difference between UserPrompt
and SystemPrompt
. It is not required to provide a SystemPrompt
string. The Model
value is mandatory for OpenAI, but is ignored for Azure OpenAI type configurations where it is implicitly specified by the deployment already.
For specific models it is possible to force the assistant response to be a valid JSON structure using the optional ENUM_ResponseFormat_Chat
parameter; if no value is specified, the default value as specified in the OpenAI documentation will be assumed by the API.
Input parameters
Name | Type | Mandatory | Description |
---|---|---|---|
UserPrompt |
String | mandatory | A user message is the input from a user. |
SystemPrompt |
String | optional | A system message can be used to specify the assistant persona or give the model more guidance and context. |
Configuration |
Configuration | mandatory | This is an object that contains endpoint and API key. |
Model |
String | only mandatory for OpenAI | This is the ID of the model to use; not considered for Azure OpenAI configurations. |
ResponseFormat |
ENUM_ResponseFormat_Chat | optional | This can be used to specify the format that the model must output. |
Return value
Name | Type | Description |
---|---|---|
AssistantResponseText |
String | An assistant message was generated by the model as a response to a user message. |
4.3.1.2 Chat Completions (with History)
Use the microflow ChatCompletions_Execute_WithHistory
to execute a chat completions API call with a ChatCompletionsSession input and a string output of the assistant response. It is not required to provide a SystemPrompt
string. The Model
value is mandatory for OpenAI, but is ignored for Azure OpenAI type configurations where it is implicitly specified by the deployment already. For certain models it is possible to force the assistant response to be a valid JSON structure using the optional ENUM_ResponseFormat_Chat
parameter; if no value is specified, the default value as specified in the OpenAI documentation will be assumed by the API.
Input parameters
Name | Type | Mandatory | Description |
---|---|---|---|
ChatCompletionsSession |
String | mandatory | This is a wrapper object for a list of messages comprising the conversation so far. |
Configuration |
Configuration | mandatory | This is an object that contains endpoint and API key. |
Model |
String | only mandatory for OpenAI | This is the ID of the model to use; not considered for Azure OpenAI configurations. |
ResponseFormat |
ENUM_ResponseFormat_Chat | optional | This can be used to specify the format that the model must output. |
Return value
Name | Type | Description |
---|---|---|
AssistantResponseText |
String | Assistant message that was generated by the model as a response to a user message. |
4.3.1.3 Chat Completions (Advanced)
For developers who want to configure the ChatCompletionsRequest object themselves and adjust its attributes according to their needs, Mendix recommends using the ChatCompletions_CallAPI
microflow. The inputs and output are shown in the table below:
Input parameters
Name | Type | Mandatory | Description |
---|---|---|---|
ChatCompletionsRequest |
ChatCompletionsRequest | mandatory | This is the request object with associated messages as specified by the Chat Completions API. |
Configuration |
Configuration | mandatory | This is the an object that contains endpoint and API key. |
Return value
Name | Type | Description |
---|---|---|
ChatCompletionsResponse |
ChatCompletionsResponse | This is the response object containing the assistant message and other details about the request. |
This option can be used if the default values of the ChatCompletionsRequest
are insufficient and must be changed to work for your specific use case. It is also useful if you are interested in other ChatCompletionsResponse values apart from the assistant response like usage metrics or multiple choices.
The following flows may be used in order to construct and handle the required inputs: ChatCompletionsRequest_Create
, ChatCompletionsMessages_Create
and ChatCompletionsMessageRequest_Create
.
4.3.2 Image Generations
The image generations API from OpenAI accepts a JSON structure that consists of a number of parameters including the user prompt as input and generates a structure of one or many model-generated images as output. The image is returned as a URL or as a base64 string. Depending on the model used, the API can return one or many model-generated images based on the input prompt plus other optional parameters. The exposed microflows in this connector are built to abstract away part of the complexity of the input and output structures and are meant to facilitate easier implementation in certain use cases. Currently, only the OpenAI API provides support for images (not Azure OpenAI).
4.3.2.1 Image Generations (Single Image)
Use the microflow ImageGenerations_Execute
to execute a single image generations API call based on a prompt string input, where the response is mapped as an image onto the OutputImage
object. The OutputImage
instance must be a specialization of GeneratedImage
. It is not required to provide the Model
, ENUM_Size
, UserString
, ENUM_Quality
, ENUM_Style
and ENUM_ResponseFormat_Image
values. For the optional parameters, if left empty, the default value as specified by the OpenAI documentation will be assumed in the API.
Input parameters
Name | Type | Mandatory | Description |
---|---|---|---|
OutputImage |
specialization of GeneratedImage | mandatory | This is the target instance in which the resulting image will be stored. It must be instantiated before calling this flow, and it must be a custom specialization of the GeneratedImage entity. |
Prompt |
String | mandatory | This is the prompt that is used by the model to generate the image. |
Configuration |
Configuration | mandatory | This is an object that contains endpoint and API key. |
Model |
String | only mandatory for OpenAI | This is the ID of the model to use. This is not considered for Azure OpenAI configurations. |
Size |
ENUM_Size | optional | This can be used to request a specific image size. The default value is 1024x1024 . |
Quality |
ENUM_Quality | optional | This is the quality of the image that will be generated. This param is only supported for dall-e-3. |
Style |
ENUM_Style | mandatory | This is the style of the generated images. This param is only supported for dall-e-3. |
ResponseFormat |
ENUM_ResponseFormat_Image | mandatory | This is the format in which the generated images are returned. Must be one of url or b64_json. Defaults to url. |
UserString |
String | optional | This is a unique identifier representing your end-user, which can help OpenAI to monitor and detect abuse. |
Return value
Name | Type | Description |
---|---|---|
IsSuccess |
Boolean | The value is true if the image generations request was successful. The value is false if an error occurred or a validation failed. |
4.3.2.2 Image Generations (advanced)
For developers who want to configure the ImageGenerationsRequest object themselves and adjust its attributes according to their needs, Mendix recommends using the ImageGenerations_CallAPI
microflow. The inputs and output are shown in the table below:
Input parameters
Name | Type | Mandatory | Description |
---|---|---|---|
ImageGenerationsRequest |
ImageGenerationsRequest | mandatory | This is the request object for the Image Generations API. |
Configuration |
Configuration | mandatory | This is an object that contains endpoint and API key. |
Return value
Name | Type | Description |
---|---|---|
ImageGenerationsResponse |
ImageGenerationsResponse | This is the response object containing the generated image. |
The microflow ImageGenerationsRequest_Create
may be used here to create and handle the input request in a custom way.
4.3.3 Embeddings
The embeddings API from OpenAI accepts a complex JSON structure that consists of a number of parameters plus one or more text strings as input and generates a structure of model-generated vector embeddings as output; per input string one vector is returned. Depending on the use case, there may be a need of generating an embedding for a single text at a time, whereas in the case of processing larger amount of data, bigger texts or data sets will be split up in discrete chunks, for which embeddings can be generated using batches of multiple input texts. The exposed microflows in this connector are built to abstract away the complex message structure and are meant to facilitate easier implementation in certain use cases.
4.3.3.1 Embeddings (single input)
Use the microflow Embeddings_Execute_SingleInput
to execute a call to the embeddings API for a single string input. The output will be the string representation of a vector embedding for the input. See ENUM_EncodingFormat_Embeddings for information of what is suported in terms of vector encoding formats. The encoding format can be left empty: if no value is specified, the default value as specified in the OpenAI documentation will be assumed by the API. The Model
value is mandatory for OpenAI, but is ignored for Azure OpenAI type configurations where it is implicitly specified by the deployment already.
Input parameters
Name | Type | Mandatory | Description |
---|---|---|---|
Input |
String | mandatory | This is the input text to embed. |
Configuration |
Configuration | mandatory | This is an object that contains endpoint and API key. |
Model |
String | only mandatory for OpenAI | This is the ID of the model to use. This is not considered for Azure OpenAI configurations. |
EncodingFormat |
ENUM_EncodingFormat_Embeddings | optional | This can be used to specify the format in which the generated vectors must be returned. |
Return value
Name | Type | Description |
---|---|---|
EmbeddingVector |
String | This is the string representation of a vector embedding for the input. |
4.3.3.2 Embeddings (list input)
Use the microflow Embeddings_Execute_ListInput
to execute an embeddings API call with a DataBatch input with a list of text strings, attached to the batch in the form of DataChunk objects. The resulting embedding vectors returned by the model end up in the EmbeddingVector
string attribute of the DataChunks. See ENUM_EncodingFormat_Embeddings for the information of what encoding formats are supported. The encoding format can be left empty: if no value is specified, the default value as specified in the OpenAI documentation will be assumed by the API. The Model
value is mandatory for OpenAI, but is ignored for Azure OpenAI type configurations where it is implicitly specified by the deployment already.
Input parameters
Name | Type | Mandatory | Description |
---|---|---|---|
DataBatch |
DataBatch | mandatory | This is a wrapper object for a list of DataChunk objects with Inputs for which an Embeddings vector should be generated. |
Configuration |
Configuration | mandatory | This is an object that contains endpoint and API key. |
Model |
String | only mandatory for OpenAI | This is the ID of the model to use. This is not considered for Azure OpenAI configurations. |
EncodingFormat |
ENUM_EncodingFormat_Embeddings | optional | This can be used to specify the format in which the generated vectors must be returned. |
Return value
Name | Type | Description |
---|---|---|
Success |
Boolean | The value is true if the embeddings request was successful. The value is false if an error occurred or a validation failed. |
The DataBatch
is a wrapper object for the list of text strings for which the embeddings are generated. You can use DataBatch_Create
to create a new Databatch
and with DataChunk_Create
new DataChunk
objects will be added to the wrapper. The order is not relevant technically here; each DataChunk
will be enriched with the corresponding embedding vector that was returned in the API call: the microflow Embeddings_Execute_ListInput
already takes care of mapping the result onto the correct DataChunk
entities and the microflow itself only returns a Success
Boolean.
4.3.3.3 Embeddings (Advanced)
For developers who want to configure the EmbeddingsRequest object themselves and adjust its attributes according to their needs, Mendix recommends using the Embeddings_CallAPI
microflow. The inputs and output are shown in the table below:
Input parameters
Name | Type | Mandatory | Description |
---|---|---|---|
EmbeddingsRequest |
EmbeddingsRequest | mandatory | This is the request object for the Embeddings API. |
Configuration |
Configuration | mandatory | This is an object that contains endpoint and API key. |
Return value
Name | Type | Description |
---|---|---|
EmbeddingsResponse |
EmbeddingsResponse | This is the response object containing the generated embedding vectors. |
This option can be used if the default values and behavior of the EmbeddingsRequest
are insufficient and must be changed to work for your specific use case. It is also useful if you are interested in other EmbeddingsResponse values apart from the vector embeddings, like usage metrics.
The following flows may be used in order to construct and handle the required inputs: EmbeddingsRequest_Create
and EmbeddingsInput_Create
.
5 Showcase Application
For more inspiration or guidance on how to use those microflows in your logic, Mendix highly recommends downloading the showcase app from the Marketplace that displays a variety of example use cases.
6 Troubleshooting
6.1 Outdated JDK Version Causing Errors while Calling the Embeddings API
The Java Development Kit (JDK) is a framework needed by Mendix Studio Pro to deploy and run applications. For more infomation, see Studio Pro System Requirements. Usually, the right JDK version is installed during the installation of Studio Pro, but in some cases it still may be outdated causing exceptions when calling the Embeddings API or other REST-based services with large data volumes.
We have seen the following two exceptions when using JDK version jdk-11.0.3.7-hotspot
:
java.net.SocketException - Connection reset
or
javax.net.ssl.SSLException - Received fatal alert: record_overflow
.
Follow these steps to check your JDK version and update if neccessary:
- Check your JDK version: In Studio Pro Go to Edit -> Preferences -> Deployment-> JDK directory. If the path points to
jdk-11.0.3.7-hotspot
, you need to update the JDK by following the next steps. - Go to Eclipse Temurin JDK 11 and download the
.msi
file of the latest release of JDK 11. - Open the downloaded file and follow the installation steps. Remember the installation path. Usually this should be somethink like
C:/Program Files/Eclipse Adoptium/jdk-11.0.22.7-hotspot
. - After the installation has finished, you might be asked to restart your computer.
- Open Studio Pro and go to Edit -> Preferences -> Deployment -> JDK directory. Click on Browse and select the folder with the new JDK version you just installed. This should be the folder containing the bin folder. Save your settings by clicking OK.
- Run the project and execute the action that threw the above mentioned execption earlier.
- You might get an error saying
FAILURE: Build failed with an exception. The supplied javaHome seems to be invalid. I cannot find the java executable.
. Verify that you have selected the right JDK directory containing the updated JDK version. You may also need to updated Gradle. For this, go to Edit -> Preferences -> Deployment -> Gradle directory. Click Browse and select a newer gradle version from the Mendix folder. In this case we replacedgrade-7.6
withgradle-7.6.3
. Save your settings by clicking OK. - Rerun the project.
- You might get an error saying
7 Read More
- Prompt Engineering – Open AI Documentation
- Introduction to Prompt Engineering – Microsoft Azure Documentation
- Prompt Engineering Techniques – Microsoft Azure Documentation
- ChatGPT Prompt Engineering for Developers - DeepLearning.AI
Feedback
Was this page helpful?
Glad to hear it! Thank you for your response.
Sorry to hear that. Please tell us how we can improve.