Multimodal User Interactions

What Are Multimodal User Interactions?

The term “multimodal interaction” refers to the ability to interact with information using multiple sensory modalities, like sound and text. For example, if a teacher provides information to students with both an audio recording and an image simultaneously, that teacher is using a multimodal interaction with his students.

On the input side, the human brain takes in information through multiple senses such as sight, sound, and touch, using “input devices” such as eyes, ears, and fingers. Computers use input devices like keyboards, scanners, digital cameras, and joysticks to take in information.

When it comes to information output, schools traditionally have only supported one format at a time for students to consume information. For example, the use of text only to convey math concepts. Multimodal inputs or outputs can incorporate one or more of our senses simultaneously to convey complex concepts in science, technology, engineering, art, and math (STEAM).

Multimodal learning occurs when students are presented with more than one output to convey the same piece of information. For example, in math, teachers use manipulatives such as blocks to help teach addition. Students are presented with the equation 2+2=x. Some students in the class may have memorized the equation to know that x=4. However, it is unclear whether they really understand the mechanics behind the equation. When students use blocks to work out the answer — taking two blocks, adding two more blocks, and counting the total blocks to get the answer — they deepen their learning experience. Research has proven that that the use of tactile objects like blocks along with words and symbols in math can enhance learning for many students (Piaget, 1954). The availability of multimodal options also gives students agency over how they learn best. This is critical for students with disabilities who often have unique learning styles.

Although sometimes used interchangeably, multimodal differs from multimedia. “Multimodal” tends to be used in scholarly environments and is focused on the process of learning. “Multimedia,” on the other hand, is often used in the public sphere and focuses on delivery of information (Lauer, 2009). We differentiate the two in this report (see the chapter entitled, “Multimedia Interactives”.)

This chapter will cover some of the technologies that can deliver a single piece of information through different modes so that students can pick the mode that is most effective for them. The interactions can be between a human and a machine or between a human and a physical object (Picciano, 2009). Some tools have been in use for decades, while others are on the cusp of technological breakthroughs. These modes of information can stimulate all five senses to make sure students have the opportunities to experience information in ways that work for them.

 

Why Are Multimodal User Interactions Important?

The advantage of multimodal interaction it that allows learners to use more of their senses at once when taking in information, lending complexity to their perception. It gives students a multifaceted way to interact with new information, which may in turn stimulate new ideas and associations that are especially helpful in a learning environment. Multimodal interfaces give teachers flexibility in how they teach to different learning styles in their classrooms, especially of students with disabilities (Tzovaras, 2008).

As discussed in the Personalized Learning chapter of this report, the current trend in education is to provide learning materials to students in ways that work for them. By having access to a variety of modalities, students have options to experience the same materials in a variety of ways.

Multimodal formats are consistent with the “Universal Design for Learning” (UDL) framework to improve and optimize teaching and learning for all people based on scientific insights into how humans learn” (CAST, 2017). The UDL guidelines point to the main benefit of multimodal formats: learner engagement.

The availability of multimodal formats motivates students and stimulates learning. The use of multimodal interactions can motivate and engage a student either by enabling access to information or simply by addressing their interests and strengths. Multiple forms of representation can communicate information in a more meaningful way than a single modality can, and these same tools can help students display their understanding of certain concepts.

 

Who is Working on This?

Many companies have been producing alternative modes of accessing information for some time, and in this chapter, we can highlight only a fraction of the work that is being done to help people who learn in different ways. Nonetheless, the sections below provide an overview of the main types of modalities along with specific examples of each.

2D tactiles – 2D tactiles have been used in classrooms supporting students with visual impairments for decades, The Braille Authority of North America created standards to support literacy for tactile readers. In 2010, they released a guideline for tactile graphics that set standards for how visual information needs to be displayed to accurately convey information that is adapted for the sense of touch. This guideline specified criteria for when to use tactile graphics, principles of design, production and duplication, and quality control (BANA, 2012). Companies like National Braille Press and the American Printing House for the Blind offer hundreds of resources from tactile books to interfaces. Graphiti, developed by Orbit Technology and The American Printing House for the Blind, allows students to access on-screen graphics such as graphs, maps, floor plans, and pie charts on a dynamic touch display.

3D printing – While so-called “additive manufacturing” is not a new process, advancements in 3D printing technology have made these machines simpler and more affordable for many people. This affordability, paired with the “Maker Movement,” has brought 3D printers into many homes and schools. A 3D printer paired with a computer enables users to design and create three-dimensional objects through a layering process (Excell, 2013). Most 3D printers are compatible with industry standard formats such as STL. There are repositories, such as Thingiverse, of objects for free and for purchase. In 2015, Benetech and members of the DIAGRAM community created a quick start guide for people interested in using 3D technology in the classroom. This guide discusses things to consider when using 3D manipulatives for different populations, where to find objects for production, and considerations for creating a maker lab in a school, among other topics.

One of the things that we learned from talking to educators about using 3D technology is that they often do not know where to find materials. Local libraries, museums, and even the National Institute of Health have vast collections of files ready for production.

Resources include:

• Libre 3D – An open-source repository for 3D enthusiasts. There are free downloads and printers and other hardware for purchase
• LibraryLyna – A free library of tested 3D models for the visually impaired, as well as a free request service for models that are not available in any other medium.
• pinshape – An online community and marketplace for designers and users with free images and images for sale. Designers can share and sell their 3D printable designs, and users can download designs to print on their own.
• Shapeways – A 3D printing service and marketplace.
• Smithsonian X3D – An iconic collection of Smithsonian objects.
• Thingverse – A repository of free, open-source downloads. Several files can be customized and remixed.
• 3D Hubs, an online service that allows users to upload a design and have it created and delivered.
• youmagine – An online repository of open-source designs that users can download and print on their own.

Alternative Text (a.k.a. “Alt-text”) – Provides textual information about a digital image if the link to the picture is no longer available or broken (Rouse, 2012). This feature is critical for people who are blind and rely on screen readers to read digital content. Alt-text for images is the “first principal of web accessibility” (WebAIM, 2017) and must convey all key information including contextual cues that a person might need to understand the image.

Anyone can create alt-text for images in Microsoft Office Suite documents, including Word, Outlook, and PowerPoint. Simply right-click on an image, pick “Format Picture” from the menu, then pick “Alt text.” You then get a form that lets you name and provide a brief description of the image. When you do this, readers of digital documents who use screen-reading software will be able to hear the description you entered (as long as their hardware devices support the function).

Microsoft Word and PowerPoint programs are equipped with accessibility checkers that can help a user identify places that need alt-text. In addition to these tools, Microsoft Support offers instruction and tips for making documents more accessible.

Haptics – Haptic interactions refer to the use of touch to understand information (Robles-De-La-Torre, 2010). You likely experience haptic feedback when you touch the keypad of a smartphone – a tiny vibration tells you that you have successfully pressed a key and entered a number. This technology is widely available in game consoles, mobile devices, and virtual technology mechanisms.

As this technology advances, students with disabilities, as well as their non-disabled peers, may be able to access information like maps and graphs by receiving the haptic feedback from their phones or “touching” virtual objects. Researchers in Tokyo are even using ultrasonic waves to let users haptically interact with a holographic object using a machine they call the Hapticlone (Russon, 2016). In addition, a team at the University of Bristol has developed three-dimensional haptic shapes that hang in mid-air and can be seen and touched (Starr, 2014).

Publishers can learn more about how to make images accessible in ebooks using haptic technology in the DIAGRAM Center’s Accessible Image Sample Book.

To learn more about uses of haptic technology for accessible learning, visit the DIAGRAM Center’s haptics information page.

Sonification – This technology uses non-speech audio to convey information (Kramer, 1994). A widespread use of this technology is sonar; however, researchers are increasingly experimenting with sonification for educational consumption.

The Sonification Lab in the School of Psychology at the Georgia Institute of Technology has been doing some interesting work on how this technology conveys spatial graphs using sonification for blind students. Through the use of this technology, people can use sound attributes such as pitch, tamber, panning, and volume to map data sets to help blind students access material that was previously unavailable to them.

Richard Ladner has conducted at the University of Washington, including work on sonification applications. To learn more about the possibilities for using sound to make images accessible, see the DIAGRAM Center webinar, “Tactile Graphics with a Voice.”

(See also the chapter on “Multimedia Interactives”.)

Audio Description

While captioning can open up access to videos for people who are deaf or hard of hearing, people who are blind may also benefit from some accommodations like audio description. The Federal Communications Commission defines video description as “audio-narrated descriptions of a television program’s key visual elements. These descriptions are inserted into natural pauses in the program’s dialogue.” In 2010, President Obama signed the Twenty-First Century Communications and Video Accessibility Act that increased opportunities to access modern communications for persons with disabilities .

YouDescribe, developed by Smith-Kettlewell Eye Institute, is a free web-based platform for adding audio descriptions to YouTube content. This project was initially funded by the Department of Education and relies on crowdsourcing for video descriptions.

One other example is Project Coyote, developed by the Museum of Contemporary Art Chicago and Prime Access Consulting, which has created an open source process for creating accessible image descriptions and making those available publicly.

 

How Are Multimodal User Interfaces Applied in Education?

Educators from preschool to postsecondary have been inundated with strategies and technologies designed to help students learn. These technologies (e.g., television, microcomputers, software, and simulation programs) have met with a range of success, but most teachers still use face-to-face interactions in traditional classrooms (Picciano, 2009). In higher education, a recent study found that not only do most professors engage in multimodal practice, but they also found that it actually works for their students (Reid et al., 2016). Technological advances and the growth in personalized learning platforms have increased the need for multimodal expression of concepts.

 

Challenges and Opportunities for Students with Disabilities

Tailoring educational experiences to meet the needs of individual learners can help every student, but for students with disabilities, multimodal user interfaces can mean the difference between access to information and no access. Having materials that help students learn in ways that work for them provides great opportunities for learning and also presents challenges that teachers need to be aware of.

 

Challenges

There are two major challenges in providing opportunities for multimodal interactions for students with disabilities: discoverability and production. Both may create significant barriers in getting materials into the classroom. Many educators simply do not know where to find tactile, haptic, and sonification resources that complement their curriculum.

For example, to enhance a biology lesson, an educator may choose to use a physical model of a plant cell to augment a lecture using the model that was purchased to accompany the textbook. While this model may represent the major components of a plant cell for someone who is able to look at it visually, it may not be useful to a blind student unless the model is designed for a tactile user. In this case, the student would need the teacher to find or produce a modified version of the model in order for the tactile to convey meaningful information. Unless the teacher knows where to look for accessible versions of this model, the student will likely go without.

Another example of discovery and production challenges in providing multimodal access is in sonification. Sonification tools are built into most personalized computers (e.g. audio recording, processing, and playback). These common tools work in conjunction with free and low cost software to allow synthesis, editing, and analysis (Stock & Zancanaro, 2006). However according to Bruce Walker at the Sonification Lab, School of Psychology Georgia Institute of Technology, “there are very few widely available software tools for creating sonifications and auditory graphs” and most of the creation tools are not designed for use by a teacher or student.

In 2007, Educause reported that haptic technologies required sophisticated hardware and significant processing power (Educause, 2007). A decade later many personal devices (e.g. Android phones) have haptic technology built right in. Unfortunately, many schools have not adopted this technology and preservice graduates report that they do not feel equipped to implement these technologies in the classroom (OFFICE OF Educational Technology, 2017).

 

Opportunities

As the many resources listed in the previous section “Who is Working on This,” many organizations are creating opportunities to provide multimodal experiences to students with disabilities. Despite the challenges of availability and cost, there are many free or low-cost tools that are or soon will be available to educators. Below, we highlight some of these resources.

The DIAGRAM Center, a department of education funded project, is developing Imageshare, a resource for content creators to obtain multiple forms of an image that can be incorporated into the curriculum. Although still in its early design stage, Imageshare promises to provide access to multiple expressions of STEAM concepts for use in the classroom. Developers of Imageshare have spoken to educators and understand how difficult it is to quickly find multimodal options that are accessible and affordable.

Imageshare is both a repository of accessible modalities and a registry. The repository will house a collection of 2D, 3D, alt-text, and described videos that have been vetted and confirmed accessible. Imageshare will also link to external collections like American Printing House for the Blind, Thingiverse, National Institute for Health, and more. Items in the collection will be tagged with metadata so that educators can look in one place to find what they need rather than searching a number of sites that may or may not meet their accessibility needs.

For example, a teacher who is teaching about the circulatory system and needs alternate modes of expression can go to the site and find a captioned video to show the class as well as download a 3D file to be printed at the school. This will give the teacher options to provide additional modalities to use while teaching the subject matter.

For students who are deaf or hard of hearing, Gallaudet University has created tools to help increase visual literacy.

VL2 is a story app that helps to increase bilingual education. Users of the VL2 story app can alternate between American Sign Language and reading English. They can also view fingerspelling of the written words. Researchers have found that increased access to both languages leads to greater literacy skills in both languages.

Additional resources and libraries of educational videos with captioning (both manual and automatic) can be found on platforms like YouTube. With the increased use of multimedia for instruction, technology such as Automatic Sync Technologies’ (AST) CaptionSync provides low-cost automated captioning for increased accessibility to educational material. Content accompanied by American Sign Language video, such as that available from the Center for Accessible Technology in Sign (CATS) at Georgia Tech, gives deaf students whose primary language is American Sign Language (ASL) access to more appropriate bilingual educational materials.

While Microsoft provides some options for adding alt-text to documents, sometimes alt-text is not enough. Benetech, the DIAGRAM Center, the National Center for Accessible Media (NCAM), and Touch Graphics created the Poet Training Tool for content creators such as professional publishers and their service providers and individual authors who make their material available electronically. This web-based image description resource helps those who are creating accessible digital documents learn how and when to describe various types of images frequently found in educational content. Poet provides best-practice guidelines and exercises that build skills for writing effective image descriptions.

To address the need for easier ways to use sonification in the classroom, the Georgia Tech Sonification lab developed the Sonification Sandbox, a graphical interface that allows users to map data to auditory parameters and add context (Walker & Cothran, 2003). Using Excel or other table software, users can upload data into the Sandbox. Once the data is visual on the screen, users can map coordinates of the graph with variation in timbres, pitch ranges, volumes, and pan levels. Any computer with Java (2.0 or greater) installed and a General Midi enabled sound card can run the application.

 

Story from the Field

In 2014, Chelsea Cook was an astronomy student at Virginia Tech University. Just like her peers, she had to take advanced math courses, but unlike her peers, she could not see graphs in textbooks because Chelsea is blind. Text and audio descriptions did not give Chelsea enough information to understand complex math with saddle points and cones. She worked with her teaching assistant, disability services office, and the computer science department to develop a 3D tactile that would represent the graphs.

During Chelsea’s 2013 TEDx VirginiaTech talk, “Creating Interfaces, Creating Experiences,” she showed a video of her interaction with her teacher as he handed her a 3D model of an equation with saddle points. As Chelsea feels the object for the first time, you can hear her gasp that now “she can see the math.” By having access to the tactiles, Chelsea was able to experience the educational content that was once inaccessible to her.

While this story is an amazing example of accessibility in education for a person with a disability, the next benefit of the technology was equally impressive. A year after we met Chelsea and heard her story, we met a person who was involved in the implementation of the tactile. Through our conversation with the disability services office, we learned about two unintended benefits that no one could foresee. First of all, Chelsea was not the only person who was able to access the math in a different way. Her sighted peers reported that the 3D models also helped them better understand the material as well. Additionally, we heard that the teaching assistant, who had never had a student with a disability in his class, had forever changed the way that he taught students. He was able to think about alternative ways to teach all students and personalize the instruction to match their learning styles.

 

Conclusions / Actions

With the heightened interest in personalized learning and the increased use of technology in the classroom, we are at the beginning stages of offering multimodal opportunities for students with disabilities.

 

Educators

1. Understand your student’s individual needs for accessing information: Set up stations for exploration.
2. Familiarize yourself with emerging technologies and their implications to support diverse learners.
3. Repeat lessons in multiple modes to reinforce the learning.

 

Parents

1. Discuss with your child how they think they learn best so that you can advocate for their needs.
2. Work with your child’s teacher to find the tools that help your child learn.
3. Familiarize yourself with the technology so you can help your child at home

 

Students

1. Talk to your teacher about the ways that you learn best. Tell her if it is easier to understand information by hearing about it, seeing it, or interacting with it.
2. Ask your teacher if you can complete assignments in non-traditional ways that will help demonstrate your ability.

 

References

• 7 Things You Should Know About Haptics (PDF). EDUCAUSE Learning Initiative. 2007. Retrieved August 4, 2017, from http://net.educause.edu/ir/library/pdf/eli7029.pdf
• Braille Authority North America, BANA. (2012, February). Guidelines and Standards for Tactile Graphics. Retrieved July 31, 2017, from http://www.brailleauthority.org/tg/web-manual/index.html
• Center For Accessible Technology, CAST. About Universal Design for Learning. (2015, May 11). Retrieved July 31, 2017, from http://www.cast.org/our-work/about-udl.html – .WX9A462ZORs
• Excell, Jon. “The rise of additive manufacturing”. The Engineer. Retrieved 2013-10-30.
• Kramer, G. (1993). Auditory display: Sonification, audification, and auditory interfaces. Perseus Publishing.
• Lauer, C. (2009). Contending with terms: “Multimodal” and “multimedia” in the academic and public spheres. Computers and Composition, 26(4), 225-239, from https://pdfs.semanticscholar.org/13ef/d30487fee1ce3fad2a900bdf29dc7e0d2456.pdf
• Piaget, J. (1954). Language and thought from a genetic perspective. Acta Psychologica, 10, 51-60.
• Picciano, A. G. (2009). Blending with purpose: The multimodal model. Journal of asynchronous learning networks, 13(1), 7-18.
• Reid, G., Snead, R., Pettiway, K., & Simoneaux, B. (2016). Multimodal communication in the university: Surveying faculty across disciplines. Across the Disciplines, 13(1).
• Robles-De-La-Torre, G. (2010). International Society for Haptics: Haptic technology, an animated explanation.  org, 2.
• Rouse, M. (2012, October). What is alt text (alternative text)? – Definition from WhatIs.com. Retrieved August 14, 2017, from http://whatis.techtarget.com/definition/alt-text-alternative-text
• Russon, M. (2016, January 04). Holograms you can reach out and touch developed by Japanese scientists. Retrieved August 2, 2017, from http://www.ibtimes.co.uk/japanese-scientists-develop-holograms-you-can-reach-out-touch-1535893
• Starr, M. (2014, December 2). Ultrasound creates a haptic shape that can be seen and felt. Retrieved August 13, 2017, from https://www.cnet.com/news/ultrasound-creates-a-haptic-shape-that-can-be-seen-and-felt/
• Stock, O., & Zancanaro, M. (Eds.). (2005). Multimodal intelligent information presentation (Vol. 27). Springer Science & Business Media, from http://bit.ly/2vQyjzi
• Tzovaras, D. (Ed.). (2008). Multimodal User Interfaces: From Signals to Interaction. Springer Science & Business Media.
• Twenty-First Century Communications and Video Accessibility Act of 2010, § Pub. L. 111-260 (2010).
• Walker, B. N., & Cothran, J. T. (2003). Sonification Sandbox: A graphical toolkit for auditory graphs. Georgia Institute of Technology.
• WebAIM Alternative Text. (2017, March 29). Retrieved August 14, 2017, from http://webaim.org/techniques/alttext/

 

Resources

• U.S. Department of Education, Office of Educational Technology, Reimagining the Role of Technology in Education: 2017 National Education Technology Plan Update, Washington, D.C., 2017.

Published: 2017-08-31