AI Features For Impact
Research Summary
Students are highly interested in the potential of technology to automate and augment certain tasks and activities while still retaining some control over the process.
Students categorized activities that involved analysis or creation as game changer. They were mostly neutral towards activities that focused on remembering or understanding.
If students think that a certain activity is not as important or valuable for them, they are more likely to want it to be fully automated.
Trust was one of the key factors and participants preferred to have some control over the outcome as they can not fully trust the system.
Context
Students often spend a significant portion of their time engaging in self-study activities. The mission was to enhance students' study efficiency, effectiveness, and enjoyment by integrating LLM technologies like Chat-GPT and Langchain into note taking app. Engineers and PMs conducted foundational research to understand user-centric use cases and Jobs To Be Done (JTBD) that these technologies could address.
Action
Developed a rigorous methodology based on Bloom's taxonomy, Kano model and HITL "Human-in-the-Loop" classification. Conducted primary and secondary research to identify user needs, attitudes and preferences towards AI-assisted self study features. Then conducted user centric through activities such as card sorting with 23 participants to sorted JTBD stories based on perceived value and the level of control they wished to retain (automation vs. augmentation).
Outcome
The research revealed that students were eager to leverage technology for automating and augmenting tasks while maintaining some control. They preferred full automation for tasks deemed less valuable and viewed analysis and creation tasks as essential. This study helped discover, understand and prioritize AI-assisted features that were aligned with students' preferences and needs, ensuring a frictionless, user-centric learning experience.
Research Objective
Organization's mission was to help students study smarter, better, and have fun while doing it.
Recent advancements in AI, notably the emergence of LLM technologies such as Chat-GPT and Langchain, have opened up a world of exciting possibilities, transforming the way we learn. But with so many possible, plausible and probable futures, it became imperative that we we had to narrow down our focus to the areas with the most potential.
To tackle this, I did a comprehensive review of contemporary literature and user research data to identify key jobs-to-be done by the students. I further validated those jobs-to-be-stories using a card sort activity. This helped the organization to define their product strategy and roadmap that would help maximize the user value.
Requirements Gathering
Literature Review
Research Framework
Jobs-to-be-done Stories
Card Sorting Activity
Research Analysis
Prioritization Workshop
Research objectives:
Understand pain points and study behaviors of target users,
Identify which pain points and study behaviors have the highest impact,
Identify which which behaviors are worth automating or and which worth augmenting,
Inform the technical architecture (e.g. API, prompt engineering),
Understand emerging use cases and applications of AI to impact study behaviors,
Make product planning future-proof.
Success criteria
In consultation with the stakeholders, I defined the success criteria to align on the intended outcome and concentrate the research efforts towards a well defined goal.
JTBD stories are aligned with emerging trends, study behaviors and use cases.
JTBD stories are validated based on the user feedback.
Data to rationalize prioritization of JTBD stories for maximum value and impact.
Research methodologies:
Literature review
User research
Voice of the customer review
User feature request analysis
Card sorting
Based on past research data and literature review, I generated JTBD stories to frame user intent, goals and actions. Then, I conducted a card sort activity with 23 participants.
We wanted to explore emerging AI applications in the field of education to identify new opportunities and subsequently develop the product roadmap for upcoming year. Which means:
We had to identify the emerging opportunities to create meaningful impact
We had to validate and confirm whether those opportunities are worth pursuing
I followed a two-fold approach:Opportunities Mapping : I conducted desk research to review published papers and past research data to identify potential use cases of emerging technology for self study activities.
Feature Validation : Then, I transformed those use cases into Jobs-to-be-Done (JTBD) stories and validated them through a card sort activity.
The purpose of this process was to gain insights into potential use cases, user attitudes, perceptions, and preferences, ultimately assisting us in defining the roadmap.
A key consideration was determining the optimum level of automation to ensure it does not have detrimental effects on the learning process. More explicitly, we wanted to understand the level of agency students desired to maintain over AI-driven features, and where would they draw the line before they cede control to technology for their learning experience.
Therefore, I developed a research framework which was structured to answer two key questions related to the application and usage of of AI technologies within the context of self-study.
Which study-related activities (JTBD stories) do students perceive as most valuable and effective to support their learning outcomes (JTBD desirability)?
Among those activities (JTBD stories), which activities students prefer to automate using AI, and which activities they prefer to retain their control over (user agency)?
I love it
These activities are game changer and can elevate my learning
I expect it
These activities are essential and can help with my learning
I am neutral
These activities are not essential but can be helpful at times
I ignore it
The activities are irrelevant and can not help with my learning
I dislike It
The activities are unnecessary and could even harm my learning
Suggest options for me but let me do most of the work
Automate most of it but let me customize
Automate everything for me in the background
Key Hypotheses
Validation (closed card sort)
To validate JTBD stories using the earlier described framework and gain insights into user preferences with respect to the two research questions, I designed a card sort study. Each JTBD statement corresponding to a specific study activity was transcribed onto individual cards.
I recruited twenty-five STEM university students from U.S. These participants, represented both genders, and were aged between 18 - 24.
Participants then conducted two separate card sorts categorizing given JTBD cards into pre-defined categories:
First, based on their "Perceived Value (Desire)" (KANO model)
Then, based on how much "Control (or User Agency)" they prefer to retain (HITL - Human-in-the-loop)
Analysis & Results
Finally, I analyzed the responses by 23 participants. In order to answer the initial two research questions, I observed the distribution of responses across within the following two dimensions:JTBD desirability (Love it - Dislike it)User agency (Full control - No control)
“I decided that the stuff I found unnecessary or distracting, I don't need any control. The things I would like to have full control of include note taking and the creation of organizers such as tables and flowcharts, as that is what I find to be some of the key ways for me to grasp new information.”
"activities which require critical thinking should be done by myself, such as brainstorming topics, asking thought-provoking questions, and synthesizing what I understand. The activities that I thought could be fully automated were more clerical; creating a glossary from a text or summarizing an audio lecture don't require critical thinking and can be done much more efficiently.”
For the first question (learner's perceived value), I identified the specific JTBD stories that students predominantly classified as having "higher desirability". For the second question (learner's preference to automate), I identified the specific JTBD stories that student's were more inclined to have some "level of automation".
I noticed some emerging patterns that exhibited correlations between categories of desire and autonomy. For example, some of the activities which were most loved were also some of the activities which were least preferred for full automation.
However, I did observe a slightly positive correlation between the activities students expressed as “neutral” or willing to “ignore”, and the activities they selected as "want no control". This suggests students may be more open to full automation for tasks they view as unimportant or extraneous to their learning needs.
The insights gathered from this research were synthesized into compelling JTBD stories, which were then validated through a card sort activity involving 23 participants. Here is what I found:
Students were all about tech that can automate and supercharge certain tasks.
If they thought that a certain activity is not as valuable for them, they were more likely to want it to be fully automated.
Students categorized jobs / activities that involved analysis or creation as game changer for their learning experience.
While the ones that were all about remembering or understanding, they were not as thrilling.
Key Insights
Activities that require analysis and creation, also requires personal involvement and control.
Automation vs Augmentation
Activities that are repetitive and distracting are best suited for automation, whereas activities that require focus and active thinking are best suited for augmentation.
Human In The Loop
Trust was one of the key factors and participants preferred to have some control over the outcome as they can not fully trust the system.
Insights
65% participants preferred some control on how various concepts that could come up during brainstorming could be mapped.
52% participants felt they need some control for activities that are visual in nature
Majority of 39% participants felt brainstorming can be automated
Insights
52% participants believed that this task can be fully automated
Majority of participants 65% think they don’t need control over translating from one language to another and can be done automatically
57% feel that they need full control when identifying and classifying information by highlighting and labelling important parts
Therefore, from a broader perspective, analysis suggested that for tasks perceived as less engaging or desirable, there appears to be a viable opportunity for implementing AI interventions. At the same time, tasks that are perceived as highly desirable, but are not suitable for automation, could be considered a candidate for augmentation.
The insights gathered from this research were synthesized into compelling JTBD stories, which were then validated through a card sort activity involving 23 participants. Here is what I found:
Students were all about tech that can automate and supercharge certain tasks.
If they thought that a certain activity is not as valuable for them, they were more likely to want it to be fully automated.
Students categorized jobs / activities that involved analysis or creation as game changer for their learning experience.
While the ones that were all about remembering or understanding, they were not as thrilling.
Conclusion
Students prefer that repetitive, distracting, wasteful and less important activities should be fully automated. However, they don't trust AI for activities that involve critical analysis and creativity and prefer to have personal involvement and control over those activities versus delegating them.
Depending on the context, and students' motivation and interest, an intricate balance is required when considering AI-assisted solutions for self-study activities.
This study proposed, created and applied a framework to identify the right balance between automation vs augmentation. It further helped to define the product roadmap by identifying AI-assisted features that are aligned with students' preferences and needs, ensuring an enhanced, frictionless, user-centric learning experience.
Limitations
Card sorting was limited to single market ( US) and didn’t account for socio-cultural differences.
JTBD stories need further research to dig deeper into actual context of use.
Past user research may have gaps as it did not explore emerging use cases and applications of AI.
This study achieved its objectives. However, along the way we made certain trade-offs leading to a few limitations, such as:
Recruitment of participants was limited to United States and students' perspectives from other regions were not accounted for.
Secondary research primarily focused on high-level use cases of AI and did not deeply explore the specific context of use or associated limitations of AI.
While JTBD stories aimed to formalize and standardize the study activities, further research is required to ground these stories in real-world scenarios and use cases.
Further research is required for comprehensive understanding of potential benefits and challenges for design and implementation of self study related AI features.
Next Steps
We agreed that we are in good place to have deeper discussions around these stories and will be able to Armed with these insights and a deeper understanding of the user perception, I identified the following actionable next steps:
Organize stories into a strategic roadmap, focusing on their immediate, near-term, and future impact.
Elaborate on stories and break them down into granular features and functionalities.
Validate features and functionalities to ensure their desirability, usability, and feasibility.
Groom stories to make them ready for the next development cycle.