Joeran Beel, University of Siegen, Dept. of Electrical Engineering and Computer Science
Haley May Dixon, Trinity College Dublin, School of Computer Science and Statistics
The impact of Graphical User Interfaces (GUI) for recommender systems is a little explored area. Therefore, we conduct an empirical study in which we create, deploy, and evaluate seven different GUI variations. We use these variations to display 68.260 related-blog-post recommendations to 10.595 unique visitors of our blog. The study shows that the GUIs have a strong effect on the recommender systems’ performance, measured in click-through rate (CTR). The best performing GUI achieved a 66% higher CTR than the worst performing GUI (statist. significant with p<0.05). In other words, with a few days of work to develop different GUIs, a recommender-system operator could increase CTR notably – maybe even more than by tuning the recommendation algorithm. In analogy to the ‘unreasonable effectiveness of data’ discussion by Google and others [2, 25], we conclude that the effectiveness of graphical user interfaces for recommender systems is equally ‘unreasonable’. Hence, the recommender system community should spend more time on researching GUIs for recommender systems. In addition, we conduct a survey and find that the ACM Recommender Systems Conference has a strong focus on algorithms – 81% of all short and full papers published in 2019 and 2020 relate to algorithm development, and none to GUIs for recommender systems. We also surveyed the recommender systems of 50 blogs. While most displayed a thumbnail (86%) and had a mouseover interaction (62%) other design elements were rare. Only few highlighted top recommendations (8%), displayed rankings or relevance scores (6%), or offered a ‘view more’ option (4%).
Graphical user interfaces for recommender systems are widely ignored in recommender systems research. 81% of all short and full papers published at ACM RecSys in 2019 and 2020 relate to algorithm development and none related to GUIs for recommender systems.
We hypothesize that GUIs for recommender systems have a high impact on the performance of a recommender system.
Our goal was to identify the impact that a graphical user interface may have on the performance (measured as click-through rate) of a recommender system.
We displayed 68.260 related-blog-post recommendations to 10.595 unique visitors of our blog https://isg.beel.org.
For each user, one out of seven GUI variations was randomly choosen. The variations are displayed below.
The best performing user interfaces were variation vii (abstract text appeared on mouseover) with CTR = 0.590% and CTRUser = 0.149%, and variation vi (highlight on mouseover) with CTR = 0.578% and CTRUser = 0.156% (Figure 3). The difference was statistically not significant (p>0.05). The Title-Only variation (i) performed surprisingly well with a CTR of 0.570% and CTRUser of 0.133% -- almost as good as the two best performing variations with no statistical difference (p>0.05). The variation that displayed the rank (iv) performed worst with a CTR of 0.355% and CTRUser of 0.092%. Not much better were variations (ii) (Thumbnail-Only) and variation (v) (Top recommendation being highlighted). The ‘standard’ design with the Thumbnail and Title being displayed, and no mouseover effects (variation iii) performed mediocrely.
Comparing the best GUI variation (vii; CTR = 0.590% ) and the worst variation (iv; CTR = 0.355%) shows that the best variation is 66% more effective than the worst one (stat. sign.; p<0.05). Or, in other words, the worst variation is 40% less effective than the best one (stat. sign.; p<0.05). Even when comparing the best performing layouts (vii and vi) with the ‘standard’ recommendation layout (iii; CTR = 0.482%), there is a performance gain of around 22% in CTR and 16% in CTRUser (stat. not sign.; p>0.05).
We find it interesting to see how relatively small changes had notable positive effects on the performance (adding a simple hover effect) or negative effects (displaying a ranking). Interesting is also the relatively high performance of the Title-Only recommendations (variation (i)), which, from a design perspective, appears like the exact opposite of the best performing variation (vii) with the dynamic abstract. Our results also confirm previous findings that recommender system GUIs should utilize dynamic visualization techniques like mouseover effects to either display additional details (e.g., an abstract) or simply highlighting the currently selected recommendation.
Our findings imply that with relatively little effort – it took us a few days to implement the seven layout variations and set up the A/B test – a large gain (up to 66%) in performance may be achieved. Based on our own experience with developing recommender systems, similar gains may be achieved by optimizing algorithms . However, to achieve those gains, typically more work than just a few days (or hours) is required. Consequently, in analogy to Haley et al. from Google who researched the “unreasonable effectiveness of data” , we argue that there is a unreasonable surprising impact of graphical user interfaces on the (perceived ) effectiveness of recommender systems. That being said, algorithms and user interfaces, of course, are not mutually exclusive. The perfect recommender system will have both, a carefully selected and tuned algorithm, and a well-designed user interface. However, given the ‘algorithm mania’ in the past, we argue that recommender-system GUIs research and development should receive more attention in the future.