Robot Learning and Robot Behavior Control Software
Skilligent Visual Localization System

A recognized visual landmark Skilligent Visual Localization System is a software component which enables mobile robots or unmanned vehicles to visually localize themselves by searching for and tracking known visual landmarks using an onboard video camera. The system is designed for use with indoor mobile robots, outdoor unmanned ground vehicles (UGV), unmanned aerial vehicles (UAV) or unmanned surface vehicles (USV).

Various indoor and outdoor objects can be used as visual landmarks including pieces of furniture, pictures on a wall, room entrances with distinctive surroundings, parts of an outdoor scene, house facades, and aerial images of a terrain. In special cases, artificial landmarks can be added to the environment (e.g. attached to a room's ceiling) in order to provide mobile robots with dependable visual localization aids.

Mobile Robot Localization

The visual localization system uses a-priori information about location and orientation of known visual landmarks. The system does not automatically build maps of the environment; instead, it relies on maps provided by upper-layer systems or on maps hand-crafted by a user. The maps are pre-loaded into the system's database of maps; also, a user process might dynamically create a new map in runtime and push it down to the localization system for immediate use. All maps are stored in a map database, a software component prepackaged with Skilligent Visual Localization System.

Visual localization with a forward-looking camera

The visual localization system reports estimated position and attitude of a video camera in a global reference frame defined by a map. For navigation purposes, a user process can translate the camera pose into three-dimensional pose of the robot's base. The translation formula is straightforward and includes a product of two matrices. The method works even if the robot is equipped with a pan-tilt camera; in this case, precise information about orientation of the camera relative to the robot's base is required.

Optionally, a camera might be equipped with a pan & tilt servo mechanism. A pan/tilt mechanism enables a robot "to look around" when trying to localize itself. To achieve this, the robot would have to trigger a special behavior to rotate the camera around when a location fix is urgently required. The visual localization system does not provide this behavior; instead, an upper layer of control hierarchy is expected to initiate the behavior when location update hasn't been received in a certain period of time.

Landmarks selected for better localization with a forward-looking camera The visual localization system uses underlying Skilligent Robot Vision System for recognizing and tracking visual landmarks. A mobile robot or an unmanned vehicle needs to be equipped with at least one video camera to capture images of the environment. The video camera should be mounted on the robot's base in a way which maximizes chances of detecting visual landmarks. Forward-looking, upward-looking, downward-looking and backward-looking camera arrangements are possible. By looking at what objects can serve as landmarks, one can decide what camera arrangement is best for a particular environment. For example, a downward-looking camera arrangement makes most sense for terrain-matching localization of UAVs.

Visual localization with an upward-looking camera

An upward-looking camera arrangement features a camera pointed vertically at the ceiling. This method has shown good results as it allows for high-precision indoor robot localization (a typical localization error is on a scale of a few inches). The method implies that artificial landmarks are attached to the ceiling - unless the ceiling has its own uniquely looking visual landmarks present due to specifics of the room/building.

Visual Localization for UAVs, Terrain Matching

A video below demonstrates how Skilligent Visual Localization System can be used to solve an outdoor localization problem. The camera is mounted under the belly of an unmanned aerial vehicle (UAV). The camera is used to search for visual landmarks on the ground (a terrain matching based navigation).

Software Architecture

Software Architecture of Skilligent Visual Localization System

  • A user process subscribes to receiving location updates. The user process provides a Map ID, a unique identifier of a map pre-loaded into the map database.
  • The visual localization system instructs underlying robot vision system to start searching for and tracking known landmarks. A list of known landmarks is taken from a map specified by the user process.
  • The vision system reads image frames from a video camera. It searches for known objects on every image.
  • When a landmark object is identified on a frame, the vision system provides the visual localization system with precise three-dimensional position and orientation of the landmark in the camera's reference frame (as seen by the video camera).
  • By combining the relative positioning data with a-priori known globally-referenced landmark data stored in the map, the visual localization system determines position and orientation of the camera in the global frame.
  • The visual localization system sends a location update up to a user process.
  • The user process translates the camera's position and orientation into position and orientation of the vehicle's base. The computation typically involves calculating a product of two matrices and requires knowing precise orientation of the camera relative to the robot's base.