Haptics Playground

Touch is often omitted or viewed as unnecessary in digital learning. Lack of touch feedback limits the accessibility and multimodal capacity of digital educational content. Touchscreens with vibratory, haptic feedback are prevalent, yet this kind of feedback is often under-utilized. This work provides initial investigations into the design, development, and use of vibratory feedback within multimodal, interactive, educational simulations on touchscreen devices by learners with and without visual impairments. The objective of this work is to design and evaluate different haptic paradigms that could support interaction and learning in educational simulations. We investigated the implementation of four haptic paradigms in two physics simulations. Interviews were conducted with eight learners (five sighted learners; three learners with visual impairments) on one simulation and initial results are shared. We discuss the learner outcomes of each paradigm and how they impact design and development moving forward.

The Discovery Research K-12 program (DRK-12) seeks to significantly enhance the learning and teaching of science, technology, engineering and mathematics (STEM) by preK-12 students and teachers, through research and development of innovative resources, models and tools (RMTs). Projects in the DRK-12 program build on fundamental research in STEM education and prior research and development efforts that provide theoretical and empirical justification for proposed projects. This project will research, design, and develop adaptive accessibility features for interactive science simulations. The proposed research will lay the foundation that advances the accessibility of complex interactives for learning and contribute to solutions to address the significant disparity in science achievement between students with and without disabilities. The PhET Interactive Simulations project at the University of Colorado Boulder and collaborators at Georgia Tech, with expertise in accessible technology and design, will form the project team. The project team will conduct design-based implementation research, where adaptive accessibility features for interactive science simulations are developed through co-design with students with disabilities and their teachers. Students will include those with dyslexia, visual impairments or blindness, and students with intellectual and developmental disabilities, ranging from 5th grade through high school, and recent high school graduates. The adaptive accessibility features will be implemented within a set of PhET interactive science simulations, and allow students with disabilities to access the science simulations with alternative input devices (such as keyboards, switches, and sip-and-puff devices), alter the visual display of the simulations (changing color contrast, zoom and enlarge, and simplify), hear different auditory representations of the visual display (descriptions, sonification, and text-to-speech), and control the rate of simulated events. All features will be capable of being turned on or off and modified on-the-fly by teachers or students through a global control panel that includes curated feature sets, resulting in highly flexible, highly accessible, interactive learning resources. PhET simulations are widely used in US classrooms, evidence-based, aligned with standards, and highly engaging and effective learning resources. With the proposed highly adaptive features and supporting resources, teachers will be able to quickly adapt the PhET simulations to meet the needs of many students with disabilities, simplifying the task of creating differentiated learning opportunities for students and supporting students with disabilities to engage in collaborative learning - a foundational component of a high-quality STEM education - alongside their non-disabled peers. To research, design, and develop the adaptive features and investigate their use by students, project team members will co-teach in classrooms with students with disabilities and conduct co-design activities with students, where students engage in design thinking to help design and refine the adaptive features to meet identified accessibility needs (their own and those of their peers). In addition, interviews with individual students with and without disabilities will also be conducted, to test early prototypes of individual features, to later refine the layering of the many different features, and to ensure the presence of adaptive features does not negatively impact traditional use of the simulations. The proposed work also includes surveys of teachers and students and analysis of teacher use, to refine global control features, develop curated feature sets, and develop supporting teacher resources. The project will address key questions at the heart of educational design for students with diverse needs, including how to make adaptive features that support student achievement of specific learning goals. The project will use design-based implementation research, with significant co-designing with students with disabilities (including visual impairments, cognitive disabilities, or dyslexia), interviews, case studies, and classroom implementation to design and evaluate the accessibility features. This will inform new models and theories of learning with technology. The project will investigate: 1) How students engage with, use, and learn from adaptive accessibility features, 2) how adaptive accessibility features can be designed to layer harmoniously together in a learning resource, and 3) how to effectively support access to rich, dynamic feature controls and curated feature sets for intuitive classroom use by students and teachers. The project will produce 8 PhET simulations with adaptive accessibility features and supporting teacher resources. The foundational research knowledge will result in effective design and implementation of adaptive accessibility features through the analysis of student engagement, usability, and learning from accessible simulations. Additionally, the project will provide technical infrastructure, exemplars, and software for use by other STEM education technology developers. The project team will work together to create a deep understanding of how to design adaptive science simulations with practical, usable, effective accessibility, so that learners with diverse needs can advance their science content knowledge and participate in science practices alongside their peers. The work has great potential to transform STEM learning for students with disabilities and to make simulations more effective for all learners. Results will provide insight into the effectiveness of accessible simulation-based activities and their corresponding teacher materials in engaging students in science practices and learning in the classroom. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria. Project Outcomes: Digital interactive learning tools have greatly transformed STEM education. These tools include simulations that let students change and test scenarios in real time, helping them actively explore and understand STEM concepts. By manipulating these simulations, students can experiment, develop explanations, and learn foundational science concepts through authentic science practices. This project was led by researchers in the PhET Interactive Simulations project at the University of Colorado Boulder, known for their award-winning interactive STEM simulations used globally. We used a design-based research approach to design, develop, and implement adaptive features in these learning tools. These features include novel auditory, visual, tactile, and haptic elements, which can be combined in various ways to broaden access and flexibility. Our goal was to make these interactive simulations more accessible for learners with disabilities and to develop foundational knowledge and technical infrastructure for using adaptive features to support STEM learning for all students. Our research involved interviews and co-design processes with participants with learning disabilities, visual impairments, and those without disabilities. This inclusive design approach ensures that the adaptive features developed can meet a broad range of needs. We explored various aspects of multimodal interaction, such as auditory description and sonification, visual highlighting, haptic feedback, tangible manipulatives, computer vision for hand and object tracking, and a preferences menu to control features that adapt in real-time to user inputs. We also conducted surveys with educators to understand their perceptions of adaptive features and their use in classrooms. Key outcomes included the development and investigation of nine new adaptive features like Voicing (customizable auditory descriptions delivered through the browser), Camera Input: Hands (gesture interaction via computer vision), and Interactive Highlights (bold visual highlights of interactive objects), and more. We published 16 simulations with these adaptive features, including two new simulations, Ratio and Proportionand Quadrilateral. For example, the Quadrilateral simulation includes features such as Voicing, Pan and Zoom, Sound and Sonification, and Interactive Highlights, and supports various input types, including keyboard, switch devices, sip-and-puff systems, hand gestures recognized through computer vision, and coupling with a novel 3D printed four-sided physical manipulative.Together, these adaptive features enable this geometry simulation to support learners with a broad range of sensory, mobility, or learning challenges, and simultaneously introduce new ways to interact and collaborate through embodied learning. Additionally, we created resources to facilitate the development of adaptive features, such as the Description Design Framework for auditory descriptions, the Haptics Playground mobile app for haptic feedback designers, and tools for developers like the Phrase Builder tool and Snapshot Developer tool. We also developed an initial prototype of a multimodal digital co-design environment called Paper Playground. All simulations, infrastructure, and documentation for adaptive features have been made available as open-source software on GitHub, allowing educators and developers worldwide to access and use these resources. Our simulations, professional development materials, research findings, design guidelines, exemplars, and software have been disseminated through our extensive partner networks, high-traffic education resource websites, third-party educational resources, a free online course on designing auditory descriptions (available on Coursera), conference presentations, journal publications, and professional organizations. The project fostered four new collaborations between PhET and research groups specializing in tangible design, craft technologies, and embodied design. These collaborations resulted in multiple interdisciplinary papers, new projects, and numerous hands-on learning opportunities for graduate students. Throughout this project, we made significant advancements in designing complex multimodal features. These advancements highlight the potential of adaptive and inclusive educational tools to support learners with disabilities and enhance the learning experience for all students. Moving forward, the principles and practices developed through this project will continue to influence the design and implementation of educational technologies made by the PhET team and many others, promoting a more inclusive and accessible learning environment for everyone.