- Elizabeth Rowe
- Director of Research
- Revealing the Invisible
- TERC, Landmark College, Massachusetts Institute of Technology
- Jodi Asbell-clarke
- https://edge.terc.edu/
- Director, EdGE at TERC
- Revealing the Invisible
- TERC
- Ibrahim Dahlstrom-Hakki
- Director
- Revealing the Invisible
- Landmark College
- Kelly Paulson
- Revealing the Invisible
- TERC
Revealing the Invisible
NSF Awards: 1417967
Researchers from EdGE at TERC, Landmark College, and MIT have come together to Reveal the Invisible. As part of a collaborative partnership resulting from an NSF Ideas Lab, our team is creating tools and methods that will transform how learning can be measured. Challenged with the question, “What is the most audacious question you could answer about education with big data?, we answered “We want to watch implicit learning happen in process.”
Implicit knowledge is knowledge that can be demonstrated through activity and behaviors but is not necessarily expressed in words. Rather than relying on tests with written and verbal expressions of knowledge, our study of implicit learning draws uses methods and instruments from multiple disciplines—science education, game-based learning, educational data mining, cognitive psychology, and neuroscience—to build multimodal models of learning.
We are collecting data from neurotypical learners and learners with cognitive differences who play the physics game Impulse while wearing an eye-tracking device so we can record their gameplay and eye movements simultaneously. We then create a visualization of the game from the players’ data logs, incorporating detected behaviors (from data mining models built in previous research) that are indicators of implicit knowledge of physics in Impulse gameplay. Finally, we layer the visualization with the eye-tracking data to show what parts of the screen (objects in the game) the player is attending to while demonstrating the physics learning behaviors (or not). This allows us to study the relationship between visual attention and learning.
The data architecture we have built for this work, DataArcade, is powerful and can be repurposed to collect a variety of types of multimodal data and synchronize the streams to a precision of 10 ms, which is required for multimodal analytics of learning in a fast action game. Our team is preparing to include additional data stream, such as EEG, physiological, and other sensor data, to build comprehensive multimodal models of implicit learning.
Revealing the Invisible
NSF Awards: 1417967
Researchers from EdGE at TERC, Landmark College, and MIT have come together to Reveal the Invisible. As part of a collaborative partnership resulting from an NSF Ideas Lab, our team is creating tools and methods that will transform how learning can be measured. Challenged with the question, “What is the most audacious question you could answer about education with big data?, we answered “We want to watch implicit learning happen in process.”
Implicit knowledge is knowledge that can be demonstrated through activity and behaviors but is not necessarily expressed in words. Rather than relying on tests with written and verbal expressions of knowledge, our study of implicit learning draws uses methods and instruments from multiple disciplines—science education, game-based learning, educational data mining, cognitive psychology, and neuroscience—to build multimodal models of learning.
We are collecting data from neurotypical learners and learners with cognitive differences who play the physics game Impulse while wearing an eye-tracking device so we can record their gameplay and eye movements simultaneously. We then create a visualization of the game from the players’ data logs, incorporating detected behaviors (from data mining models built in previous research) that are indicators of implicit knowledge of physics in Impulse gameplay. Finally, we layer the visualization with the eye-tracking data to show what parts of the screen (objects in the game) the player is attending to while demonstrating the physics learning behaviors (or not). This allows us to study the relationship between visual attention and learning.
The data architecture we have built for this work, DataArcade, is powerful and can be repurposed to collect a variety of types of multimodal data and synchronize the streams to a precision of 10 ms, which is required for multimodal analytics of learning in a fast action game. Our team is preparing to include additional data stream, such as EEG, physiological, and other sensor data, to build comprehensive multimodal models of implicit learning.
-
Bridging from Zoombinis to Computational Thinking
Jodi Asbell-Clarke
-
Learning Introductory Statistics with Interactive Songs
Dennis Pearl
-
Mastering mathematical modeling with TopoMath
Kurt VanLehn
-
A Peek at iCREAT Project’s Multi-Disciplinary Courses
shamsi moussavi
-
Utilizing MentorNet to Support One-on-One Mentoring
Sonia Martinez
-
NASA Academy of Aerospace Quality
Alice Smith
5273 Views
Continue the discussion of this presentation on the Multiplex. Go to Multiplex
5273 Views
presentation
has been viewed
Related videos you might be interested in...
-
Bridging from Zoombinis to Computational Thinking
Jodi Asbell-Clarke
-
Learning Introductory Statistics with Interactive Songs
Dennis Pearl
-
Mastering mathematical modeling with TopoMath
Kurt VanLehn
-
A Peek at iCREAT Project’s Multi-Disciplinary Courses
shamsi moussavi
-
Utilizing MentorNet to Support One-on-One Mentoring
Sonia Martinez
-
NASA Academy of Aerospace Quality
Alice Smith
Ibrahim Dahlstrom-Hakki
Director
Thank you for your interest in the Revealing the Invisible project. We hope this video gave you a basic idea of the work we are pursuing in this collaborative effort. We'd be happy to answer any questions that you may have and would love to hear your thoughts on how we can further extend our tools to support your work and the work of others in the field.
Thank you,
The Revealing the Invisible Team
Louis Gross
Director and Professor
Ibrahim et al.,
Thanks for an enticing look at the possibilities to enhance learning research using eye tracking in game settings. This provides a great opportunity as well to collect data that can analyze dynamic changes as the participants become more adept at the rules for a particular game. Can you say something more about what methods you are using for data driven discovery? Are you using data mining methods, neural nets? I've heard that commercial game companies are manipulating in real-time the rules of games - so can you "experiment" with the games you are using to help tease apart what I imagine are complex responses?
Cheers,
Lou
Ibrahim Dahlstrom-Hakki
Director
Hi Lou,
Thank you for watching our video and for your questions. We share your excitement at the prospect of using this type of data to better understand and eventually adapt learning games to improve student leaning. In terms of data driven discovery, we currently have a number of detectors built based on game behavior. This work was done by the team at EdGE and was based on the hand coding of specific game strategies to establish a ground truth that served as the basis for validating the detectors. We are currently extracting new features from the eye tracking data stream and are working on analyzing those using both top-down and bottom-up approaches. We will be looking for patterns that one can predict will be consistent with an understanding of newton's laws of motion (e.g. anticipatory eye movements) and we will be looking for emergent patterns associated with the currently available behavioral detectors.
The eye tracking record will give us better insight in a player's cognitive processing but is a resource intensive endeavor. Our ultimate goal is to use this rich data to develop new detectors that can be implemented at scale using either pure game behavior or supplementing it with webcam eye data to allow us to adapt the game in realtime. We also have in place the data collection backend and are developing new visualization tools that can be used across different games to allow us to bring this level of multimodal data collection to other learning games.
Best,
Ibrahim
Louis Gross
Director and Professor
Ibrahim, thanks for the detailed response. It is fascinating that you are thinking ahead to development of new detectors, and I expect that there are many new opportunities to use these if they can be produced at scale.
Courtney Arthur
I really love the opportunity you are providing to students who often may struggle with conceptual concepts through gaming mechanisms. What successes have you had so far?
Ibrahim Dahlstrom-Hakki
Director
We certainly find that many students who typically struggle in a traditional classroom can find learning through games a good way of developing basic conceptual understanding. This is not the case for all students and a bridging activity is needed to help students translate those concepts into explicit knowledge, but for a segment of students that are typically at risk including some students from low ses backgrounds and some students with disabilities, this can be a very effective approach. As we finish up analysis of our most recently collected data, we hope to find ways to improve our ability to support STEM learning in a broader range of learners.
Joseph Reilly
Very interesting work! How easy might it be to use those visualization tools and the data collection backend with other games or virtual environments for learning? I know many of the emergent features and behaviors you detect will be specific to Impulse but I'd love to try a similar suite of tools with other activities.
Thanks, -Joe
Ibrahim Dahlstrom-Hakki
Director
Our goal is to have something in place that others can use for their own work. Currently, we've only implemented this with TERC games but the games are quite varied and we've designed things with an eye to allow them to be used more broadly. Please feel free to get in touch if you have a specific application in mind.
Dave Barnes
Associate Executive Director
Elizabeth, Ibrahim, and team,
Very interesting! I'm intrigued. I'm not sure what to ask, but it seems that there might be opportunities to look at the actions of novice players as compared with experts and that dichotomy might increase learning of participants.
Ibrahim Dahlstrom-Hakki
Director
Hi Dave,
Absolutely, that is the direction we'd like to take. One of the challenges we need to address is domain expertise as opposed to game expertise as we tease apart players who understand the learning concepts but struggle with game mechanics and players who find strategies to succeed in the game without developing an understanding of the underlying concepts. The game is designed to minimize the distance between those two things (i.e. game mechanics and learning mechanics are aligned) but despite that we still see players exhibiting both of those behaviors.
Ibrahim
Further posting is closed as the showcase has ended.