Stories from Our Engineers: Lily



Welcome to the second entry in Zupt Autonomous Products and Technologies (ZAPT) blog series: Stories from Our Engineers.

As a technology company, our engineers have combined their knowledge from mechanical, navigation, localization, perception, and electrical engineering fields to build our first fully autonomous product, a robotic commercial lawnmower known as Nomad. During this series, several of our lead engineers will share their take on how they played a role in designing Nomad, what makes it unique to them, and the future they see. The next engineer we would like you to meet is Lily, one of the Navigation Engineers responsible for perception - object detection, LiDAR map matching, and stereo imagery depth and perception.

Pictured: Matt (Left) and Lilly(Right) detected by Nomad

What She Brings

Lilly has an extensive background in object and obstacle detection using cameras and LiDAR. While completing her doctorate in Mechanical Engineering, she conducted in-depth experiments using localization and perception algorithms to compute object detection at high accuracy. She says that “working on Nomad is a chance to implement what I have learned by bringing the academic scene into the real world.” Our object detection algorithm for Nomad is a version of Yolo, a famous object detection system that can run in real-time, with TensorRT to accelerate the model interference. Lilly is excited to integrate her deep understanding in this niche area of knowledge to ensure a safe, fully autonomous, robotic lawnmower.

What She Does

Lilly focuses on developing localization and perception algorithms using point cloud and imaging sensors such as LiDAR and Cameras. Our localization uses LiDAR to scan and implement a real-time set of aiding observations to the integrated solution (Kalman) when our GNSS (GPS) system is degraded due to trees or buildings. Essentially Lilly works on how Nomad can navigate very precisely within its challenging environment by allowing it to “see” and know where to mow.

Lilly has three primary responsibilities: designing Lidar SLAM algorithms, designing camera-based obstacle detection algorithms based on deep learning, and designing LiDAR-based segmentation algorithms:

1. Designing LiDAR SLAM algorithms:

There are two main modules for using LiDAR for navigation: Mapping and Localization.

In the Mapping module, if the mower enters a new area for the first time, onboard LiDAR data is collected and integrated with GNSS, resolver, and IMU measurements to build the base map to define mowing boundary and forbidden areas.

This map will be post-processed to correct small failures and transformed into a global reference frame (UTM). The steps involved are:

  • Build a base map using SLAM techniques in real-time.

  • Post-processing the map to add loop constraints, plane constraints and remove distortion using scan-and-matching techniques and RANSAC-based plane detection methods.

  • Transformed this pre-computed base map into the navigation reference frame.

In the Localization module, after building the base map, LiDAR is used to find the precise real-time position using multi-threaded frame-to-frame and frame-to-global scan matching methods. Next, the estimated LiDAR Odometry transforms into a real-time Global frame, with the odometry and absolute positions published in UTM. This process is implemented on a GPU and runs at a very high speed. Lily says it is important that "our slam algorithm includes both real-time frame-to-frame motion estimation and frame-to-global pose estimation to increase the localization accuracy and reduce the drift.”

2. Designing camera-based obstacle detection algorithm based on deep learning:

The obstacle detection function oversees identifying obstacles like people, animals, and vehicles within our environment, finding distances between the mower and obstacles, and generating warnings for the central control module and the local planner.

3. Designing LiDAR-based segmentation algorithm:

The segmentation module clusters the environment into the ground and non-ground groups. The ground segments are mow areas, and the non-ground group is the terrain that the mower cannot traverse. If the mower approaches the non-ground area, the segmentation module will warn the control module to slow down and avoid possible collisions.

What She Likes

In comparison to other companies designing robotic mowers, Nomad is unique in that the onboard sensors cover the entire surrounding 360 degrees instead of focusing only on the front – or travel direction. This makes Nomad very safe and reliable. The LiDAR we use is a MEMS spinning lidar that can detect objects up to 120 meters away. Then, the cameras take care of any possible blind spots from LiDAR coverage. Bottom line, Lilly says that “anything near our mower from any direction can be detected and if appropriate will trigger the obstacle avoidance function”.

Moreover, Nomad is designed to be adaptive to the environment. Nomad is expected to change mode with respect to its surroundings. Lilly says this adaptive function is shown “switching between fat and skinny mode when narrow access or wide spaces are detected.”

What She See’s

The vision Lilly has for the future is seeing Nomad “reliably autonomously mow in totally GNSS denied areas based purely on LiDAR navigation.” The end goal of Nomad is not only to speed up operational efficiency to mow more areas with the same amount of people safely but to mow in areas that are difficult without human intervention. With the implementation of localization and perception algorithms for object detection and LiDAR map matching, Nomad can autonomously mow challenging spaces through knowing its environment based on our sensor suite, but more importantly, using the advanced algorithms Lily has implemented.

Next time, our Stories from Our Engineers blog will feature Vash, our Control Engineer responsible for our Global Path planner and our Local path planner.


43 views0 comments

Recent Posts

See All