Binding of Isaac
In the Design Evaluation course at Simon Fraser University, our team, including Kinglsey, Eloise, and I, focused on conducting an affective evaluation of the video game "The Binding of Isaac," analyzing the subjective emotional and affective experiences of four participants. The process of user research was highly informative and rewarding, leading to remarkable outcomes and insights into the participants' gaming experiences.
My Role
In my role for the project, I took on several key responsibilities, which involved finding and booking a suitable location, ensuring that the setup was prepared for the participants' sessions, securing the necessary equipment for data collection, recording and conducting interview sessions, and transforming the collected data into coherent and readable content. Additionally, I collaborated as a co-writer in the report that presented and explained our research findings. Throughout the project, I worked alongside my teammates Kingsley and Eloise to ensure the successful execution of the affective evaluation of "The Binding of Isaac."
Problem
The project presented significant challenges as we aimed to identify the pain points of the game and reach a consensus on the research approach. However, the initial phase proved even more demanding as we encountered obstacles in securing participants, determining the most suitable research method, identifying an ideal location for conducting interviews, establishing reliable collaboration among team members, and effectively translating the collected data into a coherent and meaningful content for the final report.
Participants
Securing willing participants posed a considerable challenge, especially considering the proximity to exam time, as we required approximately one hour of their time. While numerous individuals expressed interest in participating, some were not regular video game players or were hesitant about being recorded during the evaluation process.
Location
Locating an ideal and quiet venue for the evaluation was a crucial aspect, ensuring the participants' comfort and undisturbed focus. Additionally, I aimed for a location that was conveniently accessible to the participants, minimizing the need for extensive travel.
Research method
Selecting the most appropriate research method was a pivotal decision that significantly impacted how we could break down and interpret our data, ultimately leading us to meaningful and conclusive results.
Data Agreement
Reviewing and analyzing the data posed challenges as team members might interpret the user's feedback and behaviors slightly differently, leading to differing viewpoints. Coming to a consensus within a reasonable timeframe proved to be difficult during this process.
Content
To ensure our findings were presented coherently and effectively in the final report, we faced the task of transforming raw numerical data into comprehensive and legible content. This process demanded meticulous analysis and interpretation, allowing us to convey the results in a clear and concise manner.
Goal
Our main objective for this project was to recruit willing participants, including friends and other students, who were available and open to being recorded. We chose the cued-recall debrief method as it provided a comprehensive approach to capturing the users' verbalizations and explanations while they interacted with the game. This method allowed us to gain insights into their thought processes, actions, and subsequent reflections, providing valuable information about their experiences. The interviews were conducted at the SFU library space, selected for its familiarity and proximity. During the analysis phase, we meticulously examined the results and reached a consensus using the IME scoring sheet. Finally, we transformed the raw data into comprehensive and legible content, which played a crucial role in shaping our final report.
Early Ideation
These two images showcase User 1 (top image) and User 4 (bottom image) actively engaged in gameplay while verbally expressing their thoughts and strategies. These images capture the participants' cognitive processes and decision-making during their gaming experiences.
The two images feature User 2 (top image) and User 3 (bottom image) engaging in the debrief portion after their gameplay sessions. During this phase, I conducted an in-depth interview, asking them about their thoughts, game plans, design perception, mechanics, noteworthy aspects, frustrations, and overall appeal of the game. These debrief sessions provided valuable insights into their emotional experiences and perceptions during their interaction with "The Binding of Isaac."
The Intrinsic Motivation Inventory (IMI) sheet used for the affective evaluation of "The Binding of Isaac" provided valuable insights into the participants' levels of interest, perceived competence, sense of autonomy, and emotional responses while engaging with the game. The IMI responses offered a comprehensive understanding of the users' intrinsic motivation and overall experience during their gameplay sessions.
Skills & Tools
Skills
Research
Reporting
Data Analysis
Tools
Cameras
Premiere Pro
Word
Excel
Process
Team Process
After collecting all the data, two team members conducted a rigorous evaluation and compared their findings. This process aimed to ensure the consistency of the data, particularly for the inter-rated data correlation, which formed a significant component of the project. The inter-rated data helped us assess the agreement between both evaluators
in identifying positive, negative, and neutral reactions from the participants.
To determine the reliability of the data, we calculated the r-value, and if it was found to be lower than 0.8, the data was deemed reliable for reporting purposes. This step was crucial in ensuring the accuracy and credibility of our findings during the affective evaluation of "The Binding of Isaac."
User Process
The process was standardized for all users to maintain consistent results. Firstly, each user played the game for approximately 15 minutes while their gameplay and expressions were recorded. Throughout the session, we prompted them to share their strategies and feelings about the game.
Next, we conducted a comprehensive debriefing session with each user, during which they walked us through their entire experience. They explained their decision-making process, emotions, and any other relevant thoughts that came to mind during gameplay.
Lastly, the users completed the Intrinsic Motivation Inventory (IMI) sheet, which provided valuable data on various aspects such as their interest in the game, perceived competence, feelings of autonomy and choice, and any perceived pressure or tension.
By following this standardized approach, we ensured a systematic and reliable evaluation of the users' subjective experiences during the affective evaluation of "The Binding of Isaac."
Final Designs
This video summary, skillfully crafted by Eloise, forms a significant part of our final report on the affective evaluation research. While the complete report cannot be shared due to Simon Fraser University's privacy policy, this video provides a concise glimpse into our research process and the insights we gained from the study. It encapsulates the essence of our findings and the emotional experiences of the participants during their engagement with "The Binding of Isaac."
Reflection
The project proved to be highly successful, earning us an exceptional A+ grade. Conducting research on The Binding of Isaac provided invaluable insights, uncovering issues related to poorly assigned controls, confusing user interface elements, and a feedback system that did not meet player expectations.
Throughout the project, I learned the importance of being well-prepared to address any questions that arose during user testing. By carefully considering and addressing the feedback from each tester, we were able to enhance our understanding and deliver a more refined evaluation.
One notable challenge we encountered was reaching a unanimous agreement on the IME scores. Despite our best efforts, there was one user for whom we couldn't reach a consensus due to varying interpretations from the debrief. This experience taught me the significance of open communication and the need to find effective ways to reconcile different viewpoints during the evaluation process.