1 Introduction
SAP Leonardo Machine Learning Foundation is a toolbox of intelligent technologies, services, and industry expertise provided by SAP. It helps companies transform into Intelligent Enterprises.
The Connector for SAP Leonardo Machine Learning Foundation allows you to use a number of the models which are available in the SAP Leonardo Machine Learning Foundation. The connector provides access to pre-trained and customizable models.
The models which are supported by the Connector for SAP Leonardo Machine Learning Foundation are:
- Classify Image (Customizable Image Classification)
- Classify Product from Image (Product Image Classification)
- Detect Face (Face Detection)
- Detect Topic (Topic Detection)
- Extract Image Features (Customizable Image Feature Extraction)
- Recognize Optical Character (OCR)
- Score Similarity (Similarity Scoring)
- Translate (Machine Translation)
2 Prerequisites
To use the Connector for SAP Leonardo Machine Learning Foundation, you need the following prerequisites:
- The SAP Leonardo Machine Learning Foundation connector downloaded into your app project from the Mendix Marketplace
An API key obtained from the SAP API Business Hub
Go to the SAP Leonardo Machine Learning Foundation - Functional Services page here: https://api.sap.com/package/SAPLeonardoMLFunctionalServices.
Select one of the models shown on the page:
Click Show API Key.
Click Copy Key and Close.
Use the API key in your app, as described in the next section.
3 Setting up the API Key
The Connector for SAP Leonardo Machine Learning Foundation actions are available in your microflows, once you have imported the Connector for SAP Leonardo Machine Learning Foundation module from the Mendix Marketplace.
Every call to SAP Leonardo Machine Learning Foundation needs to be authenticated by a valid API key. Instructions for obtaining this are in the Prerequisites section above.
The API key needs to be pasted as the value of the constant, APIKey which is found in Project ‘…’ > App Store modules > LeonardoMachineLearning > Constants.
4 Microflow Actions
Each action from the Connector for SAP Leonardo Machine Learning Foundation is described below.
4.1 ClassifyImage
This implements the Inference Service for Customizable Image Classification (classification) service of SAP Leonardo Machine Learning Foundation. It takes an image and detects the dominant objects present in it from a set of 1000 categories such as trees, animals, food, vehicles, people, and more. A description of the service can be found on the SAP site here: Inference Service for Customizable Image Classification.
Input
There are two inputs to this action.
File
This is an image object which is a System.Image entity, or an entity which is based on the System.Image entity.
Options
This is an object of type LeonardoMachineLearning.ImageClassifierOptions. This contains two attributes, ModelName and ModelVersion (corresponding to modelName and version in the Leonardo API), which identify a specific model to be used in the inference.
For the default inference model, and when using SAP API Business Hub, this value should be empty.
Limitations
The ClassifyImage action does not currently support a list of image objects, or images in an archive (ZIP) file.
Options are not available on the SAP API Business Hub.
Output
The ClassifyImage action returns a list of objects of entity type LeonardoMachineLearning.ImageClassificationResponse. Note that this list will contain a single object.
Entity/Attribute | Leonardo Equivalent | Description |
---|---|---|
ImageClassificationResponse | One object for each image (currently only one) | |
ImageName | name | Name of the file provided for classification (this will be empty) |
Result | results | One object per classification inferred, sorted by probability |
Label | Name of a classification inferred from the image | |
Score | Probability of the classification provided by label |
4.2 ClassifyProductFromImage
This implements the Product Image Classification API (inference_sync) service of SAP Leonardo Machine Learning Foundation. It takes an image or archive of images and classifies them into a fixed set of categories of products that are common in eCommerce. A description of the service can be found on the SAP site here: Inference Service for Product Image Classification.
Input
The input is an object which is a System.FileDocument entity, or an entity which is based on the System.FileDocument entity. For example, it could be an object based on System.Image which is a single image. It could also be an object containing a ZIP archive file containing images.
Limitations
The ClassifyProductFromImage action does not currently support a list of image objects.
Output
The ClassifyProductFromImage action returns a list of objects of entity type LeonardoMachineLearning.ProductClassificationResponse.
Entity/Attribute | Leonardo Equivalent | Description |
---|---|---|
ProductClassificationResponse | One object for each image | |
ImageName | name | Name of the file provided for classification |
Result | One object per classification inferred, sorted by probability | |
Label | Name of a classification inferred from the image | |
Score | Probability of the classification provided by label |
4.3 DetectFace
This implements the Inference Service for Face Detection (face-detection) service of SAP Leonardo Machine Learning Foundation. It takes an image and detects faces in it, if any. It returns information declaring a boundary box per face. A description of the service can be found on the SAP site here: Inference Service for Face Detection.
Input
The input is an image object which is a System.Image entity, or an entity which is based on the System.Image entity.
Limitations
The DetectFace action does not currently support a list of image objects, or images in an archive (ZIP) file.
Output
The DetectFace action returns a list of objects of entity type LeonardoMachineLearning.FaceDetectionResponse. Note that this list will contain a single object.
Entity/Attribute | Leonardo Equivalent | Description |
---|---|---|
FaceDetectionResponse | One object for each image (currently only one) | |
FaceCount | numberOfFaces | Number of faces detected in the image |
FaceCoordinates | faces | One object per face detected in the image |
Top | topmost pixel y-coordinate | |
Bottom | bottommost pixel y-coordinate | |
Left | leftmost pixel y-coordinate | |
Right | rightmost pixel y-coordinate |
4.4 DetectTopic
This implements the Inference Service for Topic Detection (topic-detection) service of SAP Leonardo Machine Learning Foundation. It detects and ranks topics in the input documents. A description of the service can be found on the SAP site here: Inference Service for Topic Detection. At least two documents must be submitted.
Input
There are two inputs to this action.
File
This is an object which is a System.FileDocument entity, or an entity which is based on the System.FileDocument entity. It must be an archive containing at least two text documents.
Options
This is an object of type LeonardoMachineLearning.TopicDetectionOptions. This contains the following attributes:
Attribute Description NumTopics Total number of topics to be detected NumTopicsPerDoc Number of most relevant topics to be listed per document NumKeywordsPerTopic Number of keywords to be listed per topic NumFeatures Maximum number of keywords to be extracted from documents (optional) All the values for the options can be found in the API documentation on the SAP API Business Hub here: https://api.sap.com/api/topic_detection_api/resource. The parameters in the API have the same names as the attributes in the TopicDetectionOptions entity.
Output
The DetectTopic action returns an object of entity type LeonardoMachineLearning.TopicDetectionResponse.
Entity/Attribute | Leonardo Equivalent | Description |
---|---|---|
TopicDetectionResponse | The object returned by the action | |
TopicDetectionResult | One object per document provided for inference | |
DocumentName | docName | Name of the file provided for inference |
TopicDetectionValue | N/A (implicit in the order of the values) | One object per topic detected |
Topic | A number identifying the topic inferred | |
Score | The ‘distance’ between the topic vector and that of the document | |
TopicKeywords | keywords | One object per keyword |
Keyword | A keyword which, together with the other keywords, defines the topic |
4.5 ExtractImageFeature
This implements the Inference Service For Customizable Image Feature Extraction (feature-extraction) service of SAP Leonardo Machine Learning Foundation. It takes an image and extracts feature vectors (measurable properties of the image such as edges or objects) for comparison, information retrieval, clustering, or further processing.
The values in the feature vector extracted from the image can be used as input to the ScoreSimilarity action. See the section on ScoreSimilarity, below.
Input
An image object which is a System.Image entity, or an entity which is based on the System.Image entity.
Limitations
The ExtractImageFeature action does not currently support a list of image objects or images in an archive (ZIP) file.
ExtractImageFeature uses only the default model for inferring vectors.
Output
The ExtractImageFeature action returns a list of objects of entity type LeonardoMachineLearning.ImageFeatureExtratorResponse [sic]. Note that this list will contain a single object.
Entity/Attribute | Leonardo Equivalent | Description |
---|---|---|
ImageFeatureExtratorResponse | One object for each image (currently only one) | |
ImageName | name | Name of the file provided for classification |
Vector | featureVector | One object per feature vector |
Value | item | Value assigned to each feature vector |
4.6 RecognizeOpticalCharacter (OCR)
This implements the Inference Service for Optical Character Recognition (OCR) service of SAP Leonardo Machine Learning Foundation. It takes one portable document file with the .pdf extension, or one image file in .jpeg, .jpe, or .png format as input. It returns the result in text or xml format.
Input
There are two inputs to this action.
Options
This is an object of type LeonardoMachineLearning.ocr_options. This contains the following attributes:
Attribute Description Default Lang The language of the text (currently only a single language can be set) en (English) OutputType The output type of the result txt PageSegMode The page segmentation mode 1 (automatic) ModelType The type of the machine learning model for ocr lstmStandard All the values for the options can be found in the API documentation on the SAP API Business Hub here: https://api.sap.com/api/ocr_api/resource?tag=OCR&path=post_ocr&method=post&opId=post_ocr. The parameters in the API have the same names as the attributes in the ocr_options entity.
File
This is an object which is a System.FileDocument entity, or an entity which is based on the System.FileDocument entity. For example, it could be an object based on System.Image which is a single image. It could also be an object containing a .pdf portable document format file.
Limitations
The RecognizeOpticalCharacter (OCR) Lang option only accepts a single language for recognition.
The ocr_options entity must be passed - the connector does not use the defaults if it is missing.
Output
The RecognizeOpticalCharacter (OCR) action returns an object of entity type LeonardoMachineLearning.OCRResponse.
Entity/Attribute | Leonardo Equivalent | Description |
---|---|---|
OCRResponse | The response object | |
PredictionValue | prediction | one or more objects depending on how the page has been segmented |
Value | Text which has been recognized in the image, returned in the requested format |
4.7 ScoreSimilarity
This implements the Inference Service For Similarity Scoring (similarity-scoring) service of SAP Leonardo Machine Learning Foundation. It compares vectors using a similarity score based on the cosine similarity of the vectors. The vectors retrieved from the ExtractImageFeature microflow action or from the Inference Service for Document Feature Extraction SAP Leonardo Machine Learning Foundation service can be used as inputs to this service.
Input
An object of entity type LeonardoMachineLearning.SimilarityScoringRequest, or an entity which is based on the LeonardoMachineLearning.SimilarityScoringRequest entity.
Attribute | Description |
---|---|
numSimilarVectors | The number of scores to return for each vector in the comparison |
VectorSets | A string, in JSON format, containing the vectors to be compared |
For each vector in VectorSets, ScoreSimilarity will return the best numSimilarVectors comparisons.
VectorSets must be a string in JSON format which, in the simplest case, contains an array (called 0) of vectors. Each vector contains an identifier and an array of values. See the example, below: example. It is also possible to compare two sets of vectors. More information is in the API documentation here: Inference Service For Similarity Scoring.
Limitations
The vectors to be scored must be provided as a string in JSON format. The action does not accept files as input.
Output
The ScoreSimilarity action returns an object of entity type LeonardoMachineLearning.SimilarityScoringResponse.
Entity/Attribute | Leonardo Equivalent | Description |
---|---|---|
SimilarityScoringResponse | One object which is the parent for the scores | |
SimilarVectorsScore | similarVectors | One object per comparison |
FirstVectorId | id | The id of the vector being compared |
SecondVectorId | id | The id of the vector which is in the top numSimilarVectors matches with FirstVectorId |
Score | A score between -1 and 1 indicating how similar the vectors are |
You have three vectors, v0, v1, and v2, containing the following values: [1.0, 2.0, 3.0, 4.0], [5.0, 6.0, 7.0, 8.0], and [1.0, 2.1, 3.1, 4.0]. In this case the JSON needs to be constructed as shown in the example, below.
{
"0": [{
"id": "v0",
"vector": [1.0, 2.0, 3.0, 4.0]
}, {
"id": "v1",
"vector": [5.0, 6.0, 7.0, 8.0]
}, {
"id": "v2",
"vector": [1.0, 2.1, 3.1, 4.0]
}]
}
If numSimilarVectors is set to 1 then the action will return just the best similarity score for each vector, and the id of the vector which is most similar.
Vector | Most Similar to | Score |
---|---|---|
v0 | v2 | 0.9998 |
v1 | v2 | 0.9705 |
v2 | v0 | 0.9998 |
4.8 Translate
This implements the Inference Service For Machine Translation (translation) service of SAP Leonardo Machine Learning Foundation. It translates one or more strings from a source language to one or more target languages. The Inference Service for Machine Translation is powered by SAP Translation Hub and is trained on SAP-related content to provide accurate domain adaptation and terminology coverage. A description of the service can be found on the SAP site here: Inference Service for Machine Translation.
Input
An object of entity type LeonardoMachineLearning.Translation. This must be associated with a number of other entities, as described below.
Entity/Attribute | Description |
---|---|
Translation | One object which is associated with target languages and text to be translated, grouped into units |
SourceLanguage | The language of the source text, identified by a two character string (for example, ‘en’ for English) |
Units | A way of grouping text to be translated |
Original Value | One or more objects containing fragments of text to be translated |
Value | A string to be translated |
Target Languages | A way of grouping the target languages for the translation |
TargetLanguagesValue | One or more objects containing a target language for the translation |
Value | A target language for the translation, identified by a two character string (for example, ‘de’ for German) |
For example, to translate the two texts Hello and Goodbye from English to both Spanish and German, you need to create objects as shown below:
Limitations
There is no validation that there is a translation model for any requested language pair. The languages supported are documented here: Inference Service for Machine Translation
Passing the key value is not supported.
Output
The Translate action returns an object of entity type LeonardoMachineLearning.TranslationResponse.
Entity/Attribute | Leonardo Equivalent | Description |
---|---|---|
TranslationResponse | One object which is the parent for the translations | |
SourceLanguage | The code indicating the language of the original text | |
TranslationResponseResult | translations | One object per text/target language combination |
TargetLanguage | language | The code indicating the target language of this translated text |
OriginalText | The original text to be translated | |
TargetText | value | The text in the target language |
5 Binding SAP Leonardo Machine Learning Foundation Services to Your App
You can use the SAP Leonardo Machine Learning (ML) Foundation Connector to access the services on SAP API Business Hub. Here, you can explore the services in an “as-simple-as-possible deployment”. However, to use the full abilities of the services, you need to bind a ML Foundation Service to your app running in SAP Business technology Platform (SAP BTP). More information on the distinction is available in the SAP blog post Production-ready integration of SAP Leonardo Machine Learning Foundation services with an SAP S/4HANA side-by-side extension built with the SAP S/4HANA Cloud SDK.
The following sections describe how to bind a ML Foundation Service to your app.
5.1 Adding ML Foundation Trial Entitlement
The ML Foundation Trial is not added automatically to your quota. Therefore, you need to add it manually.
Go to the SAP BTP cockpit for your account.
Choose the Entitlements and choose the ML Foundation Trial entitlement.
Click the Subaccount where you want to use the ML Foundation Trial services (this will usually be the trial subaccount).
You will see your subaccount overview, and can see that ML Foundation Trial has been added.
5.2 Binding an Instance of a ML Foundation Service to Your App
Go to the Space where your app is deployed on SAP BTP.
Select ml-foundation-trial-beta from the Service Marketplace
Go to Instances and click New Instance to add an instance of the service.
Choose the desired Service Plan and click Next.
Add parameters, if required, and click Next. In most cases additional parameters are not required.
Choose your app from the drop-down list of Applications. In most cases you will not need to add any parameters.
Click Next.
Type an Instance Name for the instance and click Finish.