QUALCOMM, INC. Jun 18, 2021
Qualcomm products mentioned within this post are offered by Qualcomm Technologies, Inc. and/or its subsidiaries.
Every day, millions of users in China capture and share memorable moments of their lives as short videos using Kuaishou, a popular video-sharing mobile app. Application users wait for refreshing and dynamic content, and interact with content creators in real-time and offline. For Kuaishou - which has grown to 379.2 million daily active users - providing a highly personalized and engaging experience across a growing and diverse user community is the number one priority. Satisfying demanding users is not without challenges, which is why Kuaishou is optimizing the use of AI, both on-device and in the cloud. Qualcomm Technology is working closely with Kuaishou to help deliver on the best user experience by maximizing AI compute capabilities on mobile phones powered by Qualcomm Snapdragon Mobile Platforms.
In this blog, we dive into a recent success story as a result of Qualcomm Technologies close collaboration with Kuaishou, featuring Kuaishou's popular Face Sketch as an example.
Kuaishou Face Sketch - Picture to a piece of artKuaishou's Face Sketch converts a selfie to a sketch made by a seasoned artist. At the heart of this feature is an AI-based algorithm, which plays the role of a talented and tireless artist who not only creates a sketch in a jiffy but continues to generate a sketch of every frame coming from the user's video feed.
The following diagram shows the journey of a single camera frame - while the user is taking a selfie video - processed by the Image Signal Processor (ISP) and fed to the Kwai application. Some pre-processing occurs on the frame to detect the face and crop it before it's fed into a highly customized neural network(s) (a.k.a ML algorithm) that roughly resembles a UNET. This generates the sketching effect of the face, which is blended with the input frame. This is followed by a makeup and beautification step to enhance the overall picture before displaying it via the camera preview.
Click to see a larger image
Jun 10, 2021 | 0:19
As Kuaishou continued to make improvements to this algorithm toward making the hypothetical artistmore talented, the complexity increased. More neurons and smarter connections between them obtained by training on more data, improved feature quality. However, Kuaishou soon detected that the artist could generate more compelling sketches but couldn't keep up in speed and, as a result, the frame rate for the video with this effect started dropping.
To users who record videos with this effect, this meant a choppy experience and a missed opportunity to create dynamic content to excite their viewers. The algorithm ran on a CPU by default, which is a general-purpose computing unit and not the best option to run AI workloads. Under these circumstances, one option app vendors normally adopt is to run complex AI processing in the cloud, which is subject to additional latency and variability based on network coverage. Fortunately for Snapdragon-based smartphones, there are more options - ranging from GPU to Qualcomm Hexagon Processor - with the Hexagon Processor providing the best leap in performance, coupled with power efficiency. This means smoother videos with compelling effects, while preserving battery life.
The collaboration between Qualcomm Technologies and Kuaishou began as a part of the Qualcomm Platform Solutions Ecosystem (PSE) program with the Snapdragon 765 platform. This was the first platform where moving the processing from CPU to the Hexagon Processor provided a 56 percent improvement in frame rate (frames per second) andbattery power reduction of 32 percent (discounting the display power) of the Face Sketch app, resulting in a seamless video capture experience.
To achieve this, Qualcomm Technologies and Kuaishou engineers worked together to analyze Kuaishou's requirements with respect to required performance, power considerations, scaling, and compatibility with future platform. Qualcomm Neural Processing SDK emerged naturally as the software and the Hexagon Processor as the AI accelerator. Working together on quantization was key to ensure that the performance and power uplift was achieved while preserving accuracy.
We are taking this a step further with early collaboration to ensure that this and other features under consideration are optimized for use on the latest generation Snapdragon 778G in time for the rollout of devices based on this platform. Besides the powerful features available with the updated 6th Generation Qualcomm Artificial Intelligence (AI) Engine that features the Qualcomm Hexagon 770 Processor with a fused AI accelerator, this platform features a new Qualcomm Spectra ISP capable of capturing stunning quality video that would provide the input to the Qualcomm AI Engine for running the magic algorithm. Overall, Snapdragon 778G offers the opportunity to capture and process videos at higher resolution with more complex AI algorithms, bringing out the equivalent of the work of a much more capable artist, on your smartphone.
Overall, the intent of Qualcomm's strong collaboration with Kuaishou is to significantly improve the user experience for millions of Kwai app users on Snapdragon-powered phones. And this is just the beginning.
'It's been a great collaboration with Qualcomm Technologies to push the performance boundary of the AI algorithms on Kuaishou,' said Mr. Hongchang Bian AI Research Manager at Kuaishou. 'We are excited to continue this joint effort in utilizing the power of heterogenous computing on high-end chips. By increasing the performance and improving the output quality of our neural networks, we'll be able to bring more compelling applications and value to our users.'
Learn more about the Snapdragon 778G 5G Mobile Platform
Engage with us on
Twitter and LinkedIn
Opinions expressed in the content posted here are the personal opinions of the original authors, and do not necessarily reflect those of Qualcomm Incorporated or its subsidiaries ('Qualcomm'). Qualcomm products mentioned within this post are offered by Qualcomm Technologies, Inc. and/or its subsidiaries. The content is provided for informational purposes only and is not meant to be an endorsement or representation by Qualcomm or any other party. This site may also provide links or references to non-Qualcomm sites and resources. Qualcomm makes no representations, warranties, or other commitments whatsoever about any non-Qualcomm sites or third-party resources that may be referenced, accessible from, or linked to this site.
Vidyasagar Rao
Staff Product Manager, Qualcomm Technologies
More articles from this author
About this author
Rodrigo Caruso Neves do Amaral
Marketing Communications, Qualcomm Technologies
More articles from this author
About this author
Attachments
- Original document
- Permalink
Disclaimer
Qualcomm Inc. published this content on 18 June 2021 and is solely responsible for the information contained therein. Distributed by Public, unedited and unaltered, on 19 June 2021 19:44:02 UTC.
