Skip to Main content Skip to Navigation
Journal articles

Exploring the synergy between knowledge graph and computer vision for personalisation systems

Abstract : In this paper, we explore the synergy between knowledge graph technologies and computer vision tools for personalisation systems. We propose two image user profiling approaches which map an image to knowledge graph entities representing the interests of a user who appreciates the image. The first one maps an image to entities which correspond to the objects appearing in the image. The second maps to entities which are depicted by visually similar images and which exist in the conceptual scope of the dataset within which further personalisation tasks are conducted. We show the superiority of our second approach against the baseline Google Cloud Vision API (label detection and web entity detection) in terms of accuracy metrics (precision, recall, MRR, nDCG). We also argue the importance of the capacity to create semantically useful profiles as the essence of many knowledge-or semantic-based personalisation systems is the semantic similarity calculation. We then apply our profiling approach in a novel personalisation use case where we seek to select the most appropriate images to display in recommendation banners. Our proposed knowledge-based approach tries to select the images which are the most in line with the semantic user profiles. We hypothesise that this image selection strategy allows to improve the user's perception of the recommended items. We conduct a two-stage user study with a real commercial travel dataset (1,357 package tours in 136 countries and regions depicted by 11,614 images). The results of 32 participants allow us to observe the promising performance of our approach in terms of persuasion, attention, efficiency and affinity. Abstract In this paper, we explore the synergy between knowledge graph technologies and computer vision tools for personalisation systems. We propose two image user profiling approaches which map an image to knowledge graph entities representing the interests of a user who appreciates the image. The first one maps an image to entities which correspond to the objects appearing in the image. The second maps to entities which are depicted by visually similar images and which exist in the conceptual scope of the dataset within which further personalisation tasks are conducted. We show the superiority of our second approach against the baseline Google Cloud Vision API (label detection and web entity detection) in terms of accuracy metrics (precision, recall, MRR, nDCG). We also argue the importance of the capacity to create semantically useful profiles as the essence of many knowledge-or semantic-based personalisation systems is the semantic similarity calculation. We then apply our profiling approach in a novel personalisation use case where we seek to select the most appropriate images to display in recommendation banners. Our proposed knowledge-based approach tries to select the images which are the most in line with the semantic user profiles. We hypothesise that this image selection strategy allows to improve the user's perception of the recommended items. We conduct a two-stage user study with a real commercial travel dataset (1,357 package tours in 136 countries and regions depicted by 11,614 images). The results of 32 participants allow us to observe the promising performance of our approach in terms of persuasion, attention, efficiency and affinity.
Document type :
Journal articles
Complete list of metadatas

Cited literature [32 references]  Display  Hide  Download

https://hal.sorbonne-universite.fr/hal-02187044
Contributor : Gestionnaire Hal-Su <>
Submitted on : Wednesday, July 17, 2019 - 3:41:05 PM
Last modification on : Monday, March 2, 2020 - 9:58:19 AM

File

1-s2.0-S1877050918316223-main....
Publication funded by an institution

Identifiers

Citation

Vincent Lully, Philippe Laublet, Milan Stankovic, Filip Radulovic. Exploring the synergy between knowledge graph and computer vision for personalisation systems. Procedia Computer Science, Elsevier, 2018, 137, pp.175-186. ⟨10.1016/j.procs.2018.09.017⟩. ⟨hal-02187044⟩

Share

Metrics

Record views

98

Files downloads

117