Get Started with AR

1 Introduction

Working with augmented reality (AR) is not all that different from implementing any other piece of Native functionality. Using just a few widgets, you can add two-or-three dimensional AR to your native mobile application. In this guide you will make a 3D cube by dragging and dropping widgets into your Mendix app and then configuring them.

2 Prerequisites

Before starting this guide, make sure you have completed the following prerequisites:

For Android:

• Have an Android mobile device that can support ARCore.
• Install Google Play Services for AR on your device.
• Install the latest Make It Native app on your Android mobile device for testing purposes.

For iOS:

• Have an iOS mobile device that supports ARKit (iPhone 6S and up).
• Install the latest Make It Native app on your iOS mobile device for testing purposes.

If you want to make use of the Realistic lighting option which can make your objects more lifelike, then you should also have an HDR image to use for rendering this light. Follow Babylon.JS’ guide to create a .env texture which you can use in the Container (AR) widget.

3 Embedding Widgets in Your App

Create a new app by following these steps:

1. Open Mendix Studio Pro. Select File > New App , and then select the Blank Native Mobile App.
2. Click Use this starting point.
3. Name your app Hello World and click Create app to close the dialog box.
4. Open the Home_Native page and remove the intro screen widget and other containers.
5. Download the Native Mobile AR (version 2 or higher) module from the Mendix Marketplace and import it to your app.

You will begin by embedding the following widgets into your Mendix app:

• Container (AR)
• ImageTracker (AR)
• Cube (AR)

Embed the widgets in this order:

1. Drag Container (AR) onto a native mobile page.

2. Drag ImageTracker (AR) into Container (AR).

3. Drag Cube (AR) into ImageTracker (AR):

   

Very little configuration is necessary to make your AR function. All you need is an image tracker. A tracker is an image your AR can track and put AR effects on top of. Most images can serve as a tracker, provided they have enough contrast. Complex images make for the best tracker, because they are easier for your device’s camera to track. Start by adding an image to ImageTracker (AR):

1. Double-click your ImageTracker (AR) widget.

2. Click Image > Edit.

3. Select a tracker from an image collection. This image will be the foundation of your AR application and is what your camera will track:

   

4. Set the physical size of this image tracker, this size is in meters. For example, if you printed the tracker to be 10 cm set Physical marker size to 0.1.

5. Double-click your Cube (AR) widgets.

6. Set X, Y and Z of Scale to 0.1. This way the Cube (AR) will be the same size as your physical tracker.

7. Click Run in Mendix Studio Pro to automatically refresh your Make It Native app. Now your Hello World AR app is up and running. You should now see a simple white cube rendered on top of a tracker:

   

Now that you have a simple scene running, you will a closer look at the widgets you used to understand what they do.

3.1 Container (AR)

This widget starts an AR scene: the video feed you see on AR apps. It is named Container (AR) because this widget will hold all your other AR widgets. It is the foundation of every Mendix AR mobile application:

3.2 ImageTracker (AR)

The ImageTracker (AR) widget allows you to track an image. Like Container (AR), this widget can contain other AR widgets. All widgets you put into ImageTracker (AR) will stick to the image. That means that every object (Cube (AR), Sphere (AR), and others) will follow or track the image.

The ImageTracker (AR) properties are as follows:

• Image – In the General tab, you can provide your own image to this widget by clicking Image > Edit. While it is possible to add any image, some images work better for AR tracking than others. Images can best be tracked when they have high contrast and several edges:

  

  It is possible to use a more conventional image, like a company logo, as long as it has enough contrast and edges. If your image does not posses these qualities, you will notice AR elements floating in incorrect places and motion tracking failures as you move your camera. It is also possible that your image will not be recognized at all.

• Orientation – This dictates the orientation of 3D objects on your tracker, not the orientation of the tracker itself. Leaving it on Up will make your 3D object appear right side up when you have it lying on a table.

Your standard cube will not look different when turned around. To show orientation, this tutorial has temporarily added the following texture (see the Material section below to learn how to add your own texture to an object):

Here is a cube with the Up orientation:

Here is a cube with the Left orientation:

Physical marker size - this is where you put the physical size of your tracker, and how AR determains how far away the marker is. Since most phones will use only one camera, this value is used to calculate depth. Putting a wrong value here won’t break AR, but will change the depth at which your objects are rendered. Which in turn may also lead to more jittering or ‘vibrating’ of your 3D models.

4 Configuring Your Cube (AR) Widget

The Cube (AR) widget will places a cube into the scene, specifically onto the ImageTracker (AR) widget. There are several properties you can configure in this widget to customize its behavior.

Having the Cube (AR) widget in the ImageTracker (AR) widget will render a white cube the same size as the image tracker directly on top of it. Next you will configure Cube (AR)’s properties.

4.1 General

The General tab contains basic properties for configuring position:

In the General tab you can configure Position, Rotation, and Scale (X stands for left/right, Y for up/down, and Z for depth):

• Position – the position relative to the image tracker. For example, since the position is relative to the tracker size, set X to 0.1 to make your object appear right beside the tracker instead of on top of it:

  Here is a cube in Position (0.1,0,0):

  

  Here is a cube in Position (0,0.1,0):

  

  Here is a cube in Position (0,0,0.1):

  

• Rotation – the rotation of the cube in 360 degrees. Set Rotation X to 45 to see your cube rotated 45 degrees on its X axis:

  Here is a cube with Rotation (0,0,0):

  

  Here is a cube with Rotation (45,0,0):

  

• Scale – the size of the object, also relative to the tracker it is embedded in. When the Scale is set to (0,0,0) the object will be invisible. Because 0.1 is exactly the size of the tracker, set Scale X, Y and Z to 0.05 to see your cube shrink in the tracker:

  Here is a cube with Scale (0.1,0.1,0.1):

  

  Here is a cube with Scale (0.05,0.05,0.05):

  

For Position, Rotation and Scale an Attribute can also be used to set the Position, Rotation and Scale X Y and Z values. To use the Attribute option simply select this instead of Expression at Position type, Rotation type or Scale type. The last option is to use Static, where you can also put a value but cannot change this value in the app itself.

4.2 Material

The Material tab contains properties for configuring appearance:

• Material type - can be either Texture or Color. Material type determains what will be put onto the cube, an image or a solid color.

• Texture – an image you can place on the cube by clicking Edit. The image will appear on each face of the cube. Put the example tracker here as a texture to end up with this colorful cube (note that the cube now ignores any Color value – if you want to make your cube one solid color, make sure the Texture is set to none):

  

• Color – the color of the cube. You can have either a Texture or Color, but not both. The color must be supplied in the #rrggbb format. #0CABF9 (a light blue) is the standard color. Change the value to #00FF00 to make your cube appear this way:

  

• Opacity – the clarity or opacity of the cube. 1 is fully opaque, while 0 is fully transparent. Change the color of your cube back to white and change the opacity value to 0.5 to see your cube become partly transparent:

  

• Lighting type – the way light from the scene will fall on your object.

  • Simple is the standard Lighting Type, this is not very realistic but will make the object visible and is the easiest to work with.
  • Realistic lighting is the more advanced option, it takes a reflection image into account. You have to supply this image in the Container (AR) widget (see the Prerequisites section above).

  Here is a cube with Lighting type > Simple:

  

  Here is a cube with Lighting type > Realistic:

  

4.3 Interaction

Interaction is where Mendix AR gets more advanced. These properties dictate every way you can interact with your object. Enabling any of these will also allow you to catch the events they generate in the Events tab:

• Dragging – allows a user to move an object by moving their phone (the object will follow on the floor, desk, or other surface in the middle of the screen).

• Enable dragging - dictates if the dragging behavior is enabled. For example, you could enable and disable this when the user clicks on the object using the On click event in the Events tab.

• Dragging type – specifies the type of dragging behavior:

  • Fixed Distance: makes the object follow you at always the same distance as it started
  • Fixed to world: makes the object stick to the world, for example objects like floors, desks, and walls

• On drag – an event that is triggered when the object is has started or stopped moving through dragging:

  

• Pinching – can be used to scale the object, similar to zooming in and out on a maps widget. Select Pinching > Yes, Enable pinching > true, select Enable Pinch to Scale > Yes, then pinch out on your object to examine it in detail:

  

4.4 Events

The Events tab appears this way:

Events on 3D objects work like any other event on a widget. On click is called when the object is clicked, and On hover enter is called when the object enters ‘in focus’ in the middle of the screen, On hover exit is called when the object exits that ‘focus’ in the middle of the screen.

4.5 Common

The Common tab appears this way:

The Common tab has two important properties:

• Name – is used internally in all AR Widgets and must be unique. It can be changed, but Mendix recommends keeping its generated name.

• Visible – is used to dictate the visibility of the AR component. This can be conditional just like most other widgets:

  

4.6 Appearance

The Appearance tab appears this way:

Currently the properties in the Appearance tab have no influence on AR widgets. This tab can be ignored.

5 Read More

• Native Mobile AR