NetApp : Building a data pipeline for recommender systems

There seems to be no gold standard for performing matrix factorization or designing recommender systems. Data scientists rely heavily on A/B testing to quantify the impact of these recommendations on business outcomes and to evaluate their efficacy.

With A/B testing, users can test multiple variations of ML and DL models until they find the best possible recommendation to improve their experience. Therefore, it's crucial for data scientists who are boosting the algorithms using model size and A/B testing to keep a snapshot of all data for generating models when developing recommender systems. A high-quality A/B test affects the design of the best recommendation engine to connect the right products and services to ensure customer satisfaction and to increase business revenue. The NetApp AI Control Plane was developed by keeping customer pain points in mind. It enables users to clone a namespace to seamlessly replicate datasets across regions and to perform the ML/DL model training, versioning, and A/B testing needed for developing recommender systems.

Convergence of edge computing and recommender systems. Companies are increasingly generating massive volumes of data at the network edge. In the IoT domain, recommendation functionalities are becoming essential. Recommender systems expect to deeply understand user's behavior, demand, and interest via edge servers. Mobile edge computing and AI inferencing at the edge are novel computing paradigms that are emerging to push computation and storage resources from remote servers to network edge servers. The NetApp AI inferencing at the edge solution combines multiple Lenovo edge servers with NetApp AFF storage systems, NVIDIA T4 GPUs, and ONTAP storage management capabilities to create recommender systems that are easy to deploy and manage.

NetApp cloud solutions. Many modern applications with recommender systems run in the cloud for model training. NetApp Cloud Volumes ONTAP is a highly available storage solution on public clouds that supports grow-as-you-go file shares that use NFS, CIFS, or iSCSI file services. NetApp Cloud Sync is a NetApp service for rapid and secure data synchronization. Whether you need to transfer files between on-premises NFS or SMB file shares, NetApp StorageGRID^®, NetApp ONTAP S3, or cloud services like Azure NetApp Files, Azure Blob, AWS S3, AWS EFS, Google Cloud Storage, or IBM Cloud Object Storage, Cloud Sync moves the files where you need them quickly and securely.

Data fabric, edge to core to cloud. AI workloads like recommender systems can mean resource-intensive tasks, from data management during training to managing scalable real-time AI/ML-based API endpoints. At a high level, an end-to-end AI/ML model deployment consists of three stages through which the data travels: edge, core, and cloud. This movement of data is very common in applications such as mobile apps and web apps where recommender systems are deployed.