Developers: Fast and efficient inference at the edge with Pilot AI

Developers: Fast and efficient inference at the edge with Pilot AI

Jan 25, 2021

Qualcomm products mentioned within this post are offered by Qualcomm Technologies, Inc. and/or its subsidiaries.

Throughout the current pandemic, technologies like virtual meetings, IoT edge devices, and machine learning (ML), have played a vital role in allowing society to continue moving forward. And have even compelled technologists to dream up new and innovative solutions.

One company that has been innovating and seeing new uses for their solutions during these times is Pilot AI, a computer vision (CV) company that has developed its own, light-weight yet powerful deep learning-based image recognition platform that runs at the edge.

We recently spoke to Ethan Wais, a product manager at Pilot AI, to learn more about their technology, what makes it unique, and how we are helping to power Pilot AI's platform.

A platform for the edge

Pilot AI's platform offers advanced CV analytics for the edge including person detection, pose estimation, and detection mapping (e.g., detecting people and mapping them to a floorplan).

The platform consists of two key components: an SDK with Pilot AI's own proprietary ML framework, and special pre-trained ML models developed by Pilot AI which can be retrained if needed, based on customers' needs. Many of their models are so good in fact, that they can be deployed to production systems out-of-the-box, and only need minimal tuning to adapt to customer-specific edge cases.

Ethan cites the following reasons for developing their own proprietary ML framework instead of using an off-the-shelf solution such as TensorFlow Lite:

• Ease of implementation: it was much easier to write their own, tailored algorithms based on their own math routines. This has resulted in deep learning-based CV algorithms that are so compute efficient, they can run in real time on compute-constrained edge devices while maintaining state-of-the-art accuracy.
• No external dependencies: customers don't have to worry about dependencies on external resources that could break the pipeline.
• Optimization: Pilot AI was able to implement very targeted optimizations, which often run an order of magnitude faster than other solutions on the same hardware.

A pilot project goes mainstream

Ethan says their detection mapping solutions were originally developed so retailers could better understand traffic through individual store aisles without requiring them to look at individual cameras and heatmaps. But since the onset of the pandemic, the use case has broadened to include social distancing and understanding how people move through or congregate in spaces, a process that traditionally relied on other techniques. This improved understanding has helped other types of customers (e.g., factory operators) to rearrange areas like entrances or cafeterias for improved social distancing.

From training to inference: An end-to-end solution for customers

A typical Pilot AI customer is an OEM who is developing their own IoT edge device, often equipped with just one camera, who wants to add image recognition capabilities. These devices often have compute constraints, strict security requirements (e.g., all data must be kept on the device), a need to reduce latency, and may even lack connectivity to the Internet. To determine how well the customer's use case(s) and requirements map to Pilot AI's platform capabilities, engagement with a customer starts with an initial evaluation period.

Ethan says they are extremely pleased with the results they have seen with the Qualcomm QCS610 SoC because of how the Qualcomm Hexagon DSP has added significant compute and a power/performance profile, which is designed to allow always-on AI applications for both consumer and commercial devices. Since their customers are typically interested in utilizing Hexagon to run AI models for always-on applications given the DSP's power/performance characteristics, Pilot AI has integrated and optimized their models for real-time processing using Hexagon Vector Extensions (HVX).

An additional benefit Ethan noted of working with the QCS610 is that Pilot AI and their OEM customers don't have to rely on a specialized commercial solution. OEMs can base their devices on the QCS610 with confidence in Qualcomm Technologies' industry-leading mobile chip experience and production, and this is especially beneficial for customers who produce large volumes of devices.

Ethan says their pre-trained models can be easily updated for a customer. For example, a customer can give Pilot AI some example video clips, and Pilot AI will update the model. In addition, Pilot AI's large and growing dataset means they can often deliver models that work out-of-the-box for customers. This helps to speed their customers' time to commercialization while also providing assurance their analytics work in real-world conditions.

Pilot AI works with both the OEM and Qualcomm Technologies, Inc. to integrate the platform into the customer's device firmware, and developers access its functionality through Pilot AI's C++ API. The result is an easy-to deploy solution - the OEM developer can write apps that access information (e.g., metadata) returned from the API without having to worry about issues like piping streams to the cloud for inference.

The pilot of the future

Pilot AI offers solutions spanning a range of industries including retail analytics, smart home, and government. Their use cases are now expanding to include construction site management, inventory management, fall detection using pose estimation, auto framing attendees in video conferences, and other commercial uses like detecting types of vehicles.

Here are a couple of videos to better illustrate what auto-framing can do:

We're excited to see what use cases come next and how Pilot AI solves them. We're also equally excited and proud to see that Pilot AI relies on our technologies like the QCS610 SoC, to help fulfill their customers' requirements.

For developers who want to learn more, be sure to check out the following resources:

• Open-Q 610 µSOM Development Kit
• Pilot AI Platform Overview

Qualcomm QCS610 and Qualcomm Hexagon are products of Qualcomm Technologies, Inc. and/or its subsidiaries.

AppDeveloper

Engage with us on

Twitter and LinkedIn

Opinions expressed in the content posted here are the personal opinions of the original authors, and do not necessarily reflect those of Qualcomm Incorporated or its subsidiaries ('Qualcomm'). Qualcomm products mentioned within this post are offered by Qualcomm Technologies, Inc. and/or its subsidiaries. The content is provided for informational purposes only and is not meant to be an endorsement or representation by Qualcomm or any other party. This site may also provide links or references to non-Qualcomm sites and resources. Qualcomm makes no representations, warranties, or other commitments whatsoever about any non-Qualcomm sites or third-party resources that may be referenced, accessible from, or linked to this site.

Siddhartha Franco

Product Manager, Staff

More articles from this author

About this author