Working with objects

When Mesmer crawls your app, it uses a combination of computer vision and artificial intelligence to figure out how to work with the UI on screen. After the crawl is complete, you can get a visualization of what Mesmer saw by using the object viewer. To use the object viewer:

  1. Navigate to the App Map.

  2. Click one of the screens in the tree.

  3. Click the “Show Objects” button at the top left of the workspace.

The objects identified by Mesmer will be overlaid on the screenshot.

Each object represents a point of interaction with the app, for example, a button, text box, checkbox, and so on. These are the widgets that Mesmer interacts with in order to crawl your app.

Viewing and changing object properties

To view properties about any given object, click it. The Object Library panel on the right will open with the following display:

The object properties include:

  • A picture of the object.

  • A brief description of what it is and what it does.

  • Whether or not it should be used to crawl the app.

Editing an object description

Mesmer’s AI might guess the type of object and try to populate the description for you. If you have any updates or corrections, those can be entered here by clicking the “edit” button at the top right of the pane. You will be able to provide English descriptions of what the widget is and how it’s used. Click the “save” button to commit any changes you make here.

Ignoring an object for crawling

If Mesmer falsely identifies an object, or you wish for it to not interact with it during a crawl (possibly speeding up the process), check the “Ignore Object in Crawl” checkbox.

Ordering objects

Each object on a particular screen has a numeric index that Mesmer uses during a crawl. These numbers represent the priority of the object compared to the others. When Mesmer encounters a screen during a crawl, it will prefer to work in the order defined by the objects on the screen.

The initial object ordering for a screen might not yield the best possible results for the amount time that’s allowed for a crawl. To maximize the coverage of a crawl, visiting more unique screens, increasing both the breadth and depth in the App Map, you can choose to provide some guidance to Mesmer’s AI. You can optimize the crawl results by reordering the objects in order to place more priority on a path that you believe is more likely to discover more screens in your app, given a limited amount of time for the crawl. To reorder the objects, drag and drop one object’s number indicator over top another object. The two objects will then swap numbers.

Defining new objects

If there is a point of interaction with your app that Mesmer doesn’t capture as an object, you can train it to understand that object the next time it’s encountered. To define a new object, first make sure the screenshot is already showing objects by clicking the “show objects” button in the upper left. Then, click on the screenshot in the area of the object that was not identified. It must not be obscured by another detected object. The Object Library panel will open and display an interface that lets you select the portion of the screen that contains the object, as well as describe the object.

You can move the rectangle selector by dragging it around the visible area of the screen, and resize it by dragging its anchors. Enter the object type and description, and click “update object library” to create the object, which will be used in future crawls