Introduction
As the topic of my bachelor thesis, I decided to conduct a research regarding the effects of interacting with smart recommender systems on user experience. I picked this topic because I was quite curious about how users feel about their data being processed by artificial intelligence technologies, what they expect and how they react when interacting with these systems. In addition, the lack of literature on this topic also encouraged me to work on this particular subject and learn more about AI-human interaction.
Topic
Effects of Smart Recommender Systems on the Effort and Trust Aspects of User Experience
Goal
Studying how AI-based recommender systems and data processing technologies affect the trust relationship between users and these systems and searching if the recommender systems really improve the decision-making experience by decreasing the amount of effort invested by the user.
My Role
UX Researcher
Responsibilities
Literature review, survey planning, script writing, data evaluation
Methodology
Online, unmoderated survey acquiring quantitative data from regular Netflix users between the ages of 15 to 56
Process
Firstly, I learned about how machine learning and deep learning technologies work and scanned the literature around the user experience of AI based systems. Later, considering the results from previous literature, I decided to frame my research under two aspects. Effort and Trust. I formed hypotheses before conducting the survey to test with the results.
Later, to acquire realistic data and relatable answers, I picked a platform that uses smart recommender systems. I decided on Netflix since the platform is very well known and uses smart recommendations as their brand’s strategy. I prepared a survey which collected quantitative data about both effort and trust aspects of the Netflix’s recommender systems.
To finish, I compared the findings with my hypotheses and looked for improvement options.
Research
The Survey
I wrote the questions based on previous surveys conducted on the subject. I divided the questions under 4 sections. Demographics, Effort, Trust, and Importance Ranking so that each section gathered data on the aimed subject and decrease bias. I presented 7 answering options ranging from complete negative to complete positive and one neutral option.
The Users
After sharing the survey in multiple platforms for 3 days, I received in total 310 responses.
Most of the users were from generation Z
The responses were mostly from users
that identified themselves as women
Each user indicated that they used Netflix at least once a week
The Findings
Effort
The easier users decide on what to watch, the less effort they invest to accomplish their goal of streaming a content. That is why I asked users if they could easily decide on what to watch by choosing from a recommendation. The users’ responses showed a division. Half indicated that they could decide with ease, while the other half reported that they had trouble choosing.
.png)
Most users indicated that they felt in control when using Netflix’s interface. The option that was most popular was option 7 which indicated the highest level of control. When dividing the answers under the sections “in control” and “not in control”, 81% of the users indicated that they felt in control.
Most users responded that they were satisfied with the system and liked finding items to watch via the recommender system. However, the most popular response was Option 5 which was the lowest level that leaned towards positive side, so the results demonstrated that while the users were leaning towards the positive side, they were not completely satisfied and wanted improvements. When dividing the answers under positive and negatives, 64% of users reported positive feedback.
Trust
Users showed little to no concern towards the collection and processing of their data. When asked if they fear that their data is processed and exposed, the most popular response was the Option 1 which was the highest level of the options that leaned towards the negative side. Overall, 59% of the users indicated that they did not feel uncomfortable with their data being collected, processed.
While asking about the precision and accuracy of the recommendations, I examined that most users found the recommendations interesting and suitable, however, the most popular choice was option 5 and second most popular choice was the neutral option. So, the results indicated while the items that are recommended are accurate, it did not fully meet the users’ expectations.
I asked questions to check if the users understood the reasoning behind the recommendations. While it is important to present items that users are interested, it is also important that they understand why and how an item is presented to them so that they familiarize with the system. The responses showed that a crushing majority understood why a particular item was presented to them.
After studying each aspect individually, I decided to rank the importance of all the aspects and sub aspects that I researched so that I could understand which aspect has the most and which has the least impact on the user experience. I asked the users to rank 6 sub aspects from most important to least important.
The responses showed that users thought effort was more important than trust.
The most important aspect was difficulty of choice while the least important was found to be explainability.
.png)
Takeaways
The Effort aspect turned out to be more effective on user experience compared to Trust.
The least important aspect that impacts the user experience was the explainability. Contrary to popular belief, users did not show interest on how recommendations were generated and why some particular items were presented to them.
Users’ responses indicated that as long as the recommender system improves the quality of the items that are presented and facilitate their job, they did not care much about their data being processed and collected.
Difficulty of choice, satisfaction and precision sub aspects performed worse than the rest of the sub aspects.
The most important thing that boosts the user experience was found to be the difficulty of choice. In other words, users responded positively the most when the recommender system facilitated their decision-making process.
Moving Forward
Additional research focusing on the effects of processing sensitive information such as sexuality, race, faith, home address, etc. on the user experience should be conducted and analyzed.
The effects of effort and trust aspects of recommender systems should be studied for different platforms as well. The results might alter greatly between different platforms that have different use.