QUALCOMM, INC. Machine learning is rapidly changing how software and algorithms are developed. And data is the lifeblood driving the machine learning revolution. We sat down with Roland Memisevic, senior director of engineering at Qualcomm Canada and part of Qualcomm AI Research, to get the latest updates on creating datasets at scale, data-driven AI, the latest AI research trends, the big AI challenges to overcome, and what's next in AI.
What led you to AI? Can you tell us about your work with Geoffrey Hinton, and your later academic career?
<_v3a_shape id='Text_x0020_Box_x0020_2' _o3a_gfxdata='UEsDBBQABgAIAAAAIQC75UiUBQEAAB4CAAATAAAAW0NvbnRlbnRfVHlwZXNdLnhtbKSRvU7DMBSF dyTewfKKEqcMCKEmHfgZgaE8wMW+SSwc27JvS/v23KTJgkoXFsu+P+c7Ol5vDoMTe0zZBl/LVVlJ gV4HY31Xy4/tS3EvRSbwBlzwWMsjZrlprq/W22PELHjb51r2RPFBqax7HCCXIaLnThvSAMTP1KkI +gs6VLdVdad08ISeCho1ZLN+whZ2jsTzgcsnJwldluLxNDiyagkxOquB2Knae/OLUsyEkjenmdzb mG/YhlRnCWPnb8C898bRJGtQvEOiVxjYhtLOxs8AySiT4JuDystlVV4WPeM6tK3VaILeDZxIOSsu ti/jidNGNZ3/J08yC1dNv9v8AAAA//8DAFBLAwQUAAYACAAAACEArTA/8cEAAAAyAQAACwAAAF9y ZWxzLy5yZWxzhI/NCsIwEITvgu8Q9m7TehCRpr2I4FX0AdZk2wbbJGTj39ubi6AgeJtl2G9m6vYx jeJGka13CqqiBEFOe2Ndr+B03C3WIDihMzh6RwqexNA281l9oBFTfuLBBhaZ4ljBkFLYSMl6oAm5 8IFcdjofJ0z5jL0MqC/Yk1yW5UrGTwY0X0yxNwri3lQgjs+Qk/+zfddZTVuvrxO59CNCmoj3vCwj MfaUFOjRhrPHaN4Wv0VV5OYgm1p+LW1eAAAA//8DAFBLAwQUAAYACAAAACEANC6T9/ECAADZBgAA HwAAAGNsaXBib2FyZC9kcmF3aW5ncy9kcmF3aW5nMS54bWy0Vdtu2zAMfR+wfxD03tpJmjYL6hRp thQDgjZoWvSZleVYmCxpknLr1+xb9mWjZLtNs2EFdkEAhyKpo8ND0T6/2FaSrLl1QquMdo5TSrhi OhdqmdH7u+nRgBLnQeUgteIZ3XFHL0bv353DcGnBlIIRRFBuCBktvTfDJHGs5BW4Y224wlihbQUe l3aZ5BY2iFzJpJump0kFQtHRC9RH8EBWVvwBlNTsC88noNbgEFKy4b6n4SjZ3yPDUK2vrFmYuQ3M 2fV6bonIM4rKKahQIpo0gSYNl8nBruULwLawVcjXRUG2GT0Z9AbpaZ+SXUa7ZyfdPtoRj289YZjQ Oet3ej08jIWMdPCh0ySw8uYNCFZ++j0I0qzpoLFH0ZlAUK1/rrnb1nwX6F3qLek+Vx+yid+iE0kH bxShxXCNfv+o/GfmMDTW+SuuKxKMjFrOfLxjsJ45X9NoU0JZTkuRT4WUYRECE2nJGmRGN6XwvCH+ KkuqqIcOu2rA4EEKbVF+u4iSherzXUh+xH+Uw2pkhL1zhk0FnjUD5+dgcXbQiVPob/BRSL3JKJPC UFJq+3ToC3l4jzFCyQZnMKPu6wosp0R+Vi5C+dawrfHYGmpVTTRW14ksoolnWy9bs7C6etA2H4dT MASK4VkZ9a058bjCAA424+NxtJmuDPiZWhgcsU7UO2h5t30Aa5pOeLwj13pRguG/akidG1tixiuP 6jbdqrULAen8wu8kjwMRFQ53rAI7iyTQuA1GjcEuedFYc+/qlqZtO81edFz4w7xeupcZ49jd5sba kGzxXAnhHcnV0f0CtXzCuju4LUR5UeCtq68bsgYvFPE7wwtg+H6YgBSPVlBiQGmHjrSbTtM+PsPv JO2FJ0aFZ+UUKiFx0HvoYCVYx2NTogAc/gMoc3ugYytAvuJ5mZ4iw5ppZPsmTxQOxQqq+NH3b2FK UO3gjE9sYvA8z8nK8YW5RfHquaoHCTPCKyg5eKnHrc1HKHw59tejHwAAAP//AwBQSwMEFAAGAAgA AAAhALk76UYfBwAASSAAABoAAABjbGlwYm9hcmQvdGhlbWUvdGhlbWUxLnhtbOxZS28bNxC+F+h/ WOy9sWS9YiNyYMly3MQvREqKHCmJ2mXMXS5Iyo5uRXLqpUCBtOihAXrroSgaoAEa9NIfY8BBm/6I DrkvUqLiB1wgKGwBxu7sN8PhzOzM7PDO3WcR9Y4xF4TFbb96q+J7OB6xMYmDtv9osP3Zbd8TEsVj RFmM2/4MC//uxqef3EHrI0qSIUN8PAhxhD0QFIt11PZDKZP1lRUxAjISt1iCY3g2YTxCEm55sDLm 6AQWiOjKaqXSXIkQif0NkCiVoB6Ff7EUijCivK/EYC9GEax+MJmQEdbY8VFVIcRMdCn3jhFt+yBz zE4G+Jn0PYqEhAdtv6L//JWNOytoPWOicgmvwbet/zK+jGF8tKrX5MGwWLReb9Sbm4V8DaByEddr 9Zq9ZiFPA9BoBDtNdTFlNjprna1GhjVA6aVD9lZrq1a18Ib82oLOmw31s/AalMqvL+C3t7tgRQuv QSm+sYCv11ur3bqF16AU31zAtyqbW/WWhdegkJL4aAFdaTRr3Xy3BWTC6I4Tvtaob7dWM+ElCqKh iC61xITFclmsRegp49sAUECKJIk9OUvwBI0gJruIkiEn3i4JQgi8BMVMALmyWtmu1OC/+tX1lbYI WsfI4FZ6gSZigaT08cSIk0S2/fsg1TcgZ2/fnj5/c/r899MXL06f/5qtrUVZfDsoDky+9z9988+r L72/f/vx/ctv06Xn8cLEv/vlq3d//Pkh8bDj0hRn371+9+b12fdf//XzS4f0TY6GJnxAIiy8fXzi PWQRbNChPx7yy3EMQkRMjs04EChGahWH/J4MLfT+DFHkwHWwbcfHHFKNC3hv+tRSuB/yqSQOiQ/C yALuMUY7jDut8ECtZZh5MI0D9+J8auIeInTsWruLYsvLvWkCOZa4RHZDbKl5SFEsUYBjLD31jB1h 7NjdE0Isu+6REWeCTaT3hHgdRJwmGZChFU0l0w6JwC8zl4Lgb8s2e4+9DqOuXW/hYxsJ7waiDuUH mFpmvIemEkUukQMUUdPgu0iGLiX7Mz4ycT0hwdMBpszrjbEQLp4DDvs1nP4A0ozb7Xt0FtlILsmR S+YuYsxEbrGjboiixIXtkzg0sZ+LIwhR5B0y6YLvMfsNUffgBxQvdfdjgi13n58NHkGGNVUqA0Q9 mXKHL+9hZsVvf0YnCLtSzSaPrBS7yYkzOjrTwArtXYwpOkFjjL1Hnzs06LDEsnmp9P0QssoOdgXW fWTHqrqPscCebm4W8+QuEVbI9nHAluizN5tLPDMUR4gvk7wPXjdt3oNSF7kC4ICOjkzgPoF+D+LF aZQDATKM4F4q9TBEVgFT98IdrzNu+e8i7xi8l08tNS7wXgIPvjQPJHaT54O2GSBqLVAGzABBl+FK t8Biub9kUcVVs02dfBP7pS3dAN2R1fREJD63A5rrfRr/Xe8DHcbZD68cL9v19DtuwVayumSnsyyZ 7Mz1N8tw811Nl/Ex+fibmi00jQ8x1JHFjHXT09z0NP7/vqdZ9j7fdDLL+o2bTsaHDuOmk8mGK9fT yZTNC/Q1auCRDnr02CdaOvWZEEr7ckbxrtCDHwHfM+NtICo+Pd3ExRQwCeFSlTlYwMIFHGkejzP5 BZFhP0QJTIeqvhISiEx0ILyECRgaabJTtsLTabTHxumws1pVg820sgokS3qlUdBhUCVTdLNVDvAK 8VrbQA9acwUU72WUMBazlag5lGjlRGUkPdYFozmU0Du7Fi3WHFrcVuJzVy1oAaoVXoEPbg8+09t+ ow4swATzOGjOx8pPqatz72pnXqenlxnTigBosPMIKD29pnRduj21uzTULuBpSwkj3GwltGV0gydC +AzOolNRL6LGZX29VrrUUk+ZQq8HoVWq0br9IS2u6mvgm88NNDYzBY29k7bfrDUgZEYoafsTGBrD ZZRA7Aj1zYVoAMctI8nTF/4qmSXhQm4hEaYG10knzQYRkZh7lERtX22/cAONdQ7RulVXISF8tMqt QVr52JQDp9tOxpMJHknT7QZFWTq9hQyf5grnU81+dbDiZFNwdz8cn3hDOuUPEYRYo1VVBhwTAWcH 1dSaYwKHYUUiK+NvrjBladc8jdIxlNIRTUKUVRQzmadwncoLdfRdYQPjLtszGNQwSVYIh4EqsKZR rWpaVI1Uh6VV93wmZTkjaZY108oqqmq6s5i1Ql4G5mx5tSJvaJWbGHKaWeHT1D2fctfyXDfXJxRV Agxe2M9RdS9QEAzVysUs1ZTGi2lY5eyMateOfIPnqHaRImFk/WYuds5uRY1wLgfEK1V+4JuPWiBN 8r5SW9p1sL2HEm8YVNs+HC7DcPAZXMHxtA+0VUVbVTS4gjNnKBfpQXHbzy5yCjxPKQWmllNqOaae U+o5pZFTGjmlmVOavqdPVOEUXx2m+l5+YAo1LDtgzXoL+/R/418AAAD//wMAUEsDBBQABgAIAAAA IQCcZkZBuwAAACQBAAAqAAAAY2xpcGJvYXJkL2RyYXdpbmdzL19yZWxzL2RyYXdpbmcxLnhtbC5y ZWxzhI/NCsIwEITvgu8Q9m7SehCRJr2I0KvUBwjJNi02PyRR7Nsb6EVB8LIws+w3s037sjN5YkyT dxxqWgFBp7yenOFw6y+7I5CUpdNy9g45LJigFdtNc8VZ5nKUxikkUigucRhzDifGkhrRykR9QFc2 g49W5iKjYUGquzTI9lV1YPGTAeKLSTrNIXa6BtIvoST/Z/thmBSevXpYdPlHBMulFxagjAYzB0pX Z501LV2BiYZ9/SbeAAAA//8DAFBLAQItABQABgAIAAAAIQC75UiUBQEAAB4CAAATAAAAAAAAAAAA AAAAAAAAAABbQ29udGVudF9UeXBlc10ueG1sUEsBAi0AFAAGAAgAAAAhAK0wP/HBAAAAMgEAAAsA AAAAAAAAAAAAAAAANgEAAF9yZWxzLy5yZWxzUEsBAi0AFAAGAAgAAAAhADQuk/fxAgAA2QYAAB8A AAAAAAAAAAAAAAAAIAIAAGNsaXBib2FyZC9kcmF3aW5ncy9kcmF3aW5nMS54bWxQSwECLQAUAAYA CAAAACEAuTvpRh8HAABJIAAAGgAAAAAAAAAAAAAAAABOBQAAY2xpcGJvYXJkL3RoZW1lL3RoZW1l MS54bWxQSwECLQAUAAYACAAAACEAnGZGQbsAAAAkAQAAKgAAAAAAAAAAAAAAAAClDAAAY2xpcGJv YXJkL2RyYXdpbmdzL19yZWxzL2RyYXdpbmcxLnhtbC5yZWxzUEsFBgAAAAAFAAUAZwEAAKgNAAAA AA== ' stroked='f' type='#_x0000_t202'><_v3a_textbox inset='0,0,0,0'>
<_w3a_wrap type='square'> I'm the classic case. At around 17 years old, I read an AI book by Douglas Hofstadter that really piqued my interest and got me hooked-on AI. In my mind, I was thinking of C-3PO and creating human-like robots for companions. This still excites me, and I believe that at some point in our lifetime we will at least see an accurate human interface that understands the world and can naturally communicate to us through a screen, if not an actual robot. We are going to better understand intelligence by building intelligent systems.
I got interested in neural nets around 2002 since they were a form of AI that actually seemed to work. When I decided to get a Ph.D. and pursue an academic career in neural networks, there actually were not many opportunities or funding for this research. One of the places doing leading research was Geoffrey Hinton's Toronto lab, which I was lucky enough to join. Since neural network theory was a bit messy, not so principled or based on elegant math, it was met with skepticism - you definitely needed to have an engineering mindset to deal with the randomness and exploratory nature of developing neural nets. Some of that still exists today.
What prompted you to start TwentyBN?
Around 2008, it was becoming increasingly apparent that neural networks would have a big impact with speech and a couple of years later with computer vision. The only two ingredients missing for neural networks to flourish were compute and labeled data. This was a huge surprise for me and many of my colleagues at that time.
Around 2012 I became a faculty member at MILA, a research institute in artificial intelligence in Montreal. In my MILA research, I was blown away that large, labelled data sets could work so well and felt that something was broken in the traditional machine learning workflow where researchers iterated over making model architecture changes on whatever data was available. For TwentyBN, we envisioned a workflow where you are focused more on the data rather than the model - a data-centric approach where researchers create or improve capabilities in an AI system not by tweaking architectures but by being creative around generating data. As data grows, the less important the neural network architecture becomes. Many of the AI systems we have created are computationally quite simple and run well on power-constrained edge devices, although they solve computationally fairly complex tasks. TwentyBN was created as a company with data at the center of importance.
Why was on-device AI inference an important topic for TwentyBN?
In many interactive applications, the AI needs to interface and understand the world through sensors, like cameras and microphones, and provide immediate response, so latency must be low. In addition, privacy is a big concern so processing and keeping the personal data on the device is required. At TwentyBN, we developed a fitness app where an AI coach would motivate and provide feedback as you exercised, but you certainly did not want this video footage being sent to the cloud.
Generally, since any sensory processing happens ultimately at the edge and at least some degree of processing happens to the raw sensor signal, it is fair to say that there is always some inference at the edge involved. On the other hand, there is usually some cloud component for aggregate data or compute-intensive processing. So, in practically any scenario, we are nowadays dealing with hybrid requirements, and at TwentyBN we made sure that both components - edge and cloud - were available in our applications.
You've developed unique datasets over the years. Why?
Intuitively, it seemed like the right thing to do. When you put the data sourcing into the center of everything rather than model tweaking, you are automatically pushed to ask the right questions. As a neural network researcher or developer, you need to think about the data you need rather that the data that is there. However, not many people were trying to solve this problem until quite recently - Andrew Ng's initiative with 'MLOps' is really gaining traction and creating a wakeup moment for the AI community. For TwentyBN, data sourcing evolved as a function of research needs. When the AI system is trained to solve a certain class of tasks, follow-on tasks naturally emerge, requiring new data sourcing interfaces, which then lead to new capabilities, and so no. It is a cycle.
How were you able to scale the collection of high-quality labeled data at low cost?
Operationalizing was the key from the get-go. We built the tooling for our crowd acting platform where crowd workers are paid to act out and record on video the requested concepts from researchers. This was our main focus - creating the software dedicated to this purpose of collecting data. We made it efficient and scalable, added intuitive user interfaces, and of course iterated quickly over time to respond to customer needs - including our own needs.
How can the AI research community take advantage of these datasets?
Two popular datasets we created and licensed at TwentyBN were Something Something (email here for interest) and Jester (email here for interest).
What role can these datasets play in advancing AI research happening at Qualcomm AI Research?
There are two aspects that can play a role. First, the existing datasets, like gesture control, are useful for neural network development, and second, the crowd acting platform can efficiently create new data at scale. The existing data is also useful from a transfer learning perspective. We have always used data sourcing in a 'cumulative' fashion, such that a neural network is trained on most of the data that was sourced over the years and fine-tuned on use case-specific data.
What big challenges are left to overcome in data collection?
A big challenge is on the cultural side in terms of adopting a completely different mindset and workflow to build AI systems. It is overcoming the entrenched mindset that is so common in the AI community where you tinker with the neural network architecture rather than focus on getting good data. Once you realize data is the key, it is a matter operationalizing the collection of good data and building a lot of tooling.
Another big challenge is that things don't nicely compartmentalize in AI. Figuring out what is good data is often very domain specific to the application. As a result, there is a huge benefit to vertical integration where the data, data collection, neural network design, and application are all done together. There is a very strong feedback loop when you have this end-to-end understanding and realize what data you need to keep improving. For example, the little discoveries that you make through application feedback informs the data that you need to collect.
Qualcomm Technologies recently brought you and the rest of the top-notch AI research team from TwentyBN on-board. What are your impressions of Qualcomm AI Research so far?
Qualcomm has a no-nonsense culture. There is an intellectual honesty and openness that encourages speaking the truth on technical matters without regard to politics or seniority. Technology decisions are made based on the facts. For me, that has been beautiful to see. It is definitely an engineering culture.
I'm also realizing how strong of a position Qualcomm Technologies has as an edge compute player. It is a great place to be at because the most important data is always being generated at the edge and that is where you want AI to run.
You've been working at the intersection between cutting-edge AI research and consumer products. What is the key to successfully bringing computer vision innovation to market?
What we learned at TwentyBN developing the fitness application is that you need to vertically integrate across the end-to-end stack and operationalize the data collection process to become efficient and principled.
Looking toward the future, what are the most challenging problems in the field of AI right now?
A big challenge is how do you make neural networks, which are a parallel mess, think more. AI is paradigm change in computing, where we are going from serial computing and a Von Neumann architecture to parallel processing of these big parallel messes.
I believe it makes sense to consider a 'third compute paradigm' that is much more human-like. Human brains process data in a very parallel manner unlike serial computers, yet humans have the capability to do serial thinking and reasoning as well. This is a huge research problem to understand. Humans have capabilities that are far superior to AI like creativity, common sense, and language. I believe that a key reason for these capabilities is that human symbol processing happens on a sub-symbolic substrate. Increasing the degree of 'thinking' that can happen in a sub-symbolic, parallel mess is a research area where I hope we will see a lot of progress in the coming years. While it allows us to better understand those magical aspects of human cognition, it also allows us to make better use of AI accelerator hardware than we do today.
In terms of predictions, which areas of AI research are you expecting to see large progress and exciting breakthroughs?
I predict progress in system 2 cognition, which is a deliberate type of thinking involved in focus, deliberation, reasoning or analysis, in neural networks. This goes along this third computing paradigm idea that I just mentioned.
AI will also enter our homes, transforming them into smart homes with multi-modal interaction. There is a lot of work to get this productized, but I expect a lot of progress ranging from truly smart TVs to robots.
Thanks Roland!
To learn more about our latest research, visit the Qualcomm AI Research page. And if you are interested in joining our team and making an impact at scale, please apply for one of our open machine learning positions.
Sign up for our newsletter to receive the latest information about mobile computing.
Attachments
- Original document
- Permalink
Disclaimer
Qualcomm Inc. published this content on 16 September 2021 and is solely responsible for the information contained therein. Distributed by Public, unedited and unaltered, on 16 September 2021 15:41:09 UTC.
