Evaluating Augmented Reality Frameworks for Real-time On-device Product Recognition

Status: open
Supervisor: Martin Kampel, Julian Strohmayer

Problem Statement: The ever growing product assortments of supermarkets present consumers with interesting challenges. For example, it is often difficult for health- or environmentally-conscious consumers to filter out products that are in line with their personal values from the overabundance of products. To support these consumers, mobile applications are being developed that use visual product recognition [1] [2] to highlight relevant products in the feed of a mobile phone camera. Additional product information such as ingredients, nutritional values or food certificates is often presented to the consumer in the form of an augmented reality overlay [3], as shown in Figure 1. Because product recognition itself is a computationally intensive step that takes up a large portion of the total pipeline runtime, the use of efficient augmented reality libraries is crucial when presenting product information to ensure that the developed applications can run in real time.

Goal: Evaluation of different AR frameworks for mobile devices and development of a prototype application on Android. The application should be able to track supermarket products in moving camera frames and display product information as AR overlay in real time (**Note: You do not have to perform product detection (e.g.: with a DL model), the evaluation can be based on test videos with ground truth bounding box labels).

Tasks:

  • Conducting state-of-the-art research on AR frameworks
  • Getting familiar with AR frameworks (e.g.: Google ARCore)
  • Collection of a small test dataset (videos of product shelves + bounding box labels)
  • Evaluating different AR frameworks (latency, tracking accuracy, features, practicality, …)
  • Implementation of an application prototype on Android

Keywords: Computer Vision, Augmented Reality, AR, Product Recognition

Figure 1: Example of visual product recognition with an augmented reality overlay on a mobile phone.

[1] Yuchen Wei, Son Tran, Shuxiang Xu, Byeong Kang, Matthew Springer, and Massimo Panella. 2020. Deep Learning for Retail Product Recognition: Challenges and Techniques. Intell. Neuroscience 2020 (2020). https://doi.org/10.1155/2020/8875910

[2] A. Tonioni, E. Serra and L. Di Stefano, “A deep learning pipeline for product recognition on store shelves,” 2018 IEEE International Conference on Image Processing, Applications and Systems (IPAS), 2018, pp. 25-31, doi: 10.1109/IPAS.2018.8708890.

[3] Google AI On-device Supermarket Product Recognition, https://ai.googleblog.com/2020/07/on-device-supermarket-product.html, Accessed: 11.05.2022