Multimedia Interactives

What Are Multimedia Interactives?

Multimedia interactives are applications that allow the user to “control, combine, and manipulate different types of media, such as text, sound, video, computer graphics, and animation” (Encyclopaedia Britannica, 2017).

In the paper, “A Brief Guide to Interactive Multimedia and the Study of the United States,” (n.d.) Georgetown University professor Randall Bass defines multimedia as “any package of materials that includes some combination of texts, graphics, still images, animation, video, and audio.” These materials are integrated in a way that provides multiple paths for a user to access material, in the way that works for them.

Multimedia interfaces are ubiquitous — everything from digital airport kiosks to apps on the iPhone, from talking ATMs displaying ads to video game consoles playing Minecraft – multisensory experiences abound. Multimedia interactives take that experience one step further by providing avenues for end users to manipulate their experience in real time by interacting directly with the digital media. For example, users can navigate through a forest in a Minecraft world, dissect a digital frog in science class, or use virtual turntables to mix records in a Google doodle celebrating the 44^th anniversary of hip-hop.

Increasingly, users are expecting to interact with their multimedia. Viewers of YouTube videos, for instance, click on links and other annotations to learn more information as they watch the video. Viewers of movies on the Amazon Video platform do not just watch the screen passively; with a swipe, they can instantly see a list of the actors currently onscreen while the movie is playing and learn more about their filmographies.

The most common type of interactives allow users to manipulate virtual objects, as in video games. Web browsers such as Chrome and Safari, standalone computer software, gaming consoles such as Xbox and PlayStation, and tablets such as the iPad, Amazon Fire, and Samsung Galaxy are just a few examples of the software and hardware people use to access multimedia-rich experiences.

In addition, the next generation of multimedia interactives contain two types of sophisticated interactive technologies that give the user a highly interactive virtual experience: augmented reality (AR) and virtual reality (VR). Augmented reality superimposes a computer-generated reality on the real world in order to enhance what the user sees. For example, in the interactive game Pokémon GO, players use their smartphones to “catch” fictional characters that virtually populate locations in the real world, thereby “augmenting the reality” of players. In contrast, virtual reality creates a new, immersive simulated experience unrelated to the user’s actual physical environment. For example, using a special headset or other wearable devices, users can feel what it is like to fly through the air like a superhero or wander through the orchestra during a classical concert (Vanian, 2016)

 

Why Are Multimedia Interactives Important?

Interactive multimedia has changed the landscape of how we consume content. Revenue from digital games alone is predicted to reach over $100 billion globally in 2017 (Graham, 2017). Consider that the most read New York Times article of 2013 was actually an interactive digital quiz. Numerous universities now offer interactive media and game design majors and departments, including the University of Southern California and the Rochester Institute of Technology. Consumers have demonstrated great appetite for novel interactive media experiences. For instance, in 2016 the augmented reality game Pokémon GO swept the world, eventually pulling in more than 60 million users and reaching $500 million in revenue in record time (Perez, 2016).

 

Who Is Working on This?

Multimedia interactives are appearing in a wide range of applications from a wide range of sources including college students creating immersive interactive educational games in their spare time, YouTubers creating interactive video channels, publishers enhancing their digital electronic book offerings, and Fortune 500 companies creating immersive marketing experiences. Digital media companies such as Mashable and the New York Times are adding interactive content that incorporates videos, sound clips, and interactive data visualizations to their news coverage. Fast Company’s 2017 list of the most innovative companies in VR (Terdiman, 2017) includes major technology companies such as Google, Facebook, Snap, and Sony, all of whom are investing significant resources into this new space. Specific tools and initiatives related to multimedia interactives include:

• Consumer-facing tools for creating or experiencing interactive multimedia:
  • YouTube’s interactive video tools allow users to add clickable annotations such as specially timed subscription links and elaborate choose-your-own-adventure video ads (Lopez, 2015).
  • Knight Lab’s Timeline JS – an open-source tool for creating interactive timelines.
  • Storify allows users to create web-based content that enables user interaction via in-story commenting and content tag filtering.
  • ThingLink accelerates learning by creating dynamic multimedia images, videos, and 360-degree virtual lessons.
  • Google Cardboard provides an inexpensive cardboard viewer that turns an ordinary smartphone into a VR screen when used with VR software.
  • PBS’s POV, television’s longest-running showcase for independent nonfiction films, maintains a list of 18 free and low-cost tools for making interactive video.

• Virtual and augmented reality educational platforms:
  • zSpace Learning Lab provides hardware, software, and educational content to increase the use of experimental learning in the classroom using AR and VR, especially for STEM (science, technology, engineering, and math) content.
  • Peer, a mixed reality educational experience, uses Internet-enabled sensors along with digital headsets to make abstract concepts in STEAM (science, technology, engineering, art and math) more tangible.
• Integrated learning environments (study tools, embedded audio, video, and simulations):
  • McGraw-Hill’s Connect® is an interactive online learning environment that allows teachers and students to customize the learning experience according to learning preferences.
  • Pearson’s Revel™ creates an immersive experience for students by combining subject matter, interactive media, and assessments together.
  • The University of Colorado Boulder’s PhET Interactive Simulations provide an intuitive, game-like environment where students learn through exploration and discovery. For example, in an interactive lesson about static electricity, students can experiment to see if John Travoltage will get shocked as he rubs his foot on the rug and moves his arm towards the doorknob.

• Gaming:
  • Microsoft’s Xbox with Kinect, Sony’s PlayStation, and Nintendo’s Wii all offer possibilities for education. These systems use cameras, hand controllers, helmets, and gloves to monitor students in real time as they interact with games.
  • Minecraft Education Edition – Layers tools to support collaboration and structured learning into a gaming environment.
  • Disney’s Infinity Learning – Can be used to teach game making and development with primary students.

 

How Are Multimedia Interactives Applied in Education?

With so much available content and an increasing number of tools available to create more, it is not surprising that the digital classroom is increasingly making use of multimedia interactives. Even though they are still considered in the realm of gaming, native mobile apps are now available for an enormous range of educational purposes. Numerous guides recommend education apps, from Digital Trend’s recently published “Get Smart: The 25 best educational apps for iPhone and Android” to Common Sense Media’s list of best apps for kids, which curate content across dozens of categories including best apps to learn a new language and best apps for teaching history.

Ebooks and digital learning environments from traditional educational publishers are becoming more important. According to Educause (n.d.), there has been a significant shift from printed textbooks to digital teaching and learning tools in today’s schools, and students will have ever greater control over how they interact with them. Most textbook publishing companies have launched digital platforms; in fact, several have transformed their core identities from traditional textbook publishers to learning science companies or digital education companies.

Interactive, integrated, online learning environments such McGraw-Hill’s Connect, engage students with study tools, embedded audio, video, and interactive quizzes on personal devices while giving teachers and students immediate feedback on students’ progress. Students using these media-rich interactive ebooks gain a deeper understanding of the material since they are not just reading the subject matter but experiencing it.

Multimedia interactives can also provide access to experiences that are otherwise inaccessible. While these simulated activities do not replace real-life interactions with people, places, and natural phenomena, they open a door for some students that would otherwise stay closed. For example, virtual reality platforms can help students interested in antiquities explore Greek ruins or visit archaeological treasures hidden within conflict zones without leaving their desks. Physics students can try their hand at building an atom or experimenting with semiconductors and transistors without expensive equipment.

At the University of Colorado Boulder, the PhET Interactive Simulations are based on extensive education research and engage students through an intuitive, game-like environment where students learn through exploration and discovery. For example, the PhET interactive simulation.

In sum, interactive technologies are already being used in some mainstream classrooms for educational purposes (Cuendet et al., 2013). Experts estimate that by 2025, over 15 million students will be using virtual and/or augmented reality technologies in the classroom (Bellini et al., 2016).

 

Challenges and Opportunities for Students with Disabilities

If we break apart the word multimedia into its root words, we get multiple ways to present media. A student with a disability may have limited success using a given medium. Having the content available in an alternative format may allow a student to interact with the content.

Three factors determine whether the content of multimedia interactives is accessible to users with disabilities:

1. Correct format of input into the system to control the interactive
2. Correct format of output from the system to present the interactive
3. The ability of the user to successfully operate the multimedia interactive itself

Even if all of the above factors are in place correctly from the content creator, the hardware device that the student uses may not be compatible with the given multimedia interactive.

Nevertheless, new developments in accessible multimedia interactives are emerging that show promise in overcoming the challenges.

 

Challenges

All too often multimedia interactives are made without any thought as to how accessible this media is for a person who cannot access its primary sensory output. For example, if an audio clip with no corresponding textual description is embedded in an ebook, that clip will not be accessible to a hearing-impaired student.

Interactives in general pose a number of accessibility challenges, from accessing the controls and manipulating the interactive to consuming the output from the interactive. PhET, a leader in accessible interactive simulations, knows firsthand the challenges in making simulations accessible. In their 2017 CSUN Assistive Technology Conference presentation, “Description strategies to make an interactive science simulation accessible” (Smith, Lewis and Moore), they point out a few of these challenges, including conveying unpredictable interaction sequences and conveying multiple simultaneous changes to the user.

Real-time feedback from a simulation can be too fast for students with cognitive disabilities to understand what is happening if there is no way to adjust the speed of the simulation. Depending on how the simulation’s output is presented, students who are hard of hearing may not realize that they are missing important audio feedback. Similarly, students who are visually impaired may not have access to visual output. Not only is the output of simulations a challenge, but the input method of how the student interacts with the simulation is an accessibility challenge. A number of simulations require a student to grab an object and move it around, possibly affecting the outcome of the simulation. Having an accessible drag-and-drop interactive for students with physical and visual impairments will be a challenge.

It is important to note that sometimes interactive multimedia needs to be mediated by other technology (like a web browser) or assistive technology (like a screen reader) in order to be accessible. In other words, to have an accessible multimedia interactive, both the media itself must be made accessible and the system that is used to interact with the media must also be accessible.

Here is an example of how a hardware device may render accessible content inaccessible. Imagine that you have a fully accessible ebook – all images have descriptions, audio clips have textual descriptions, videos are captioned and described, all the content is in the correct reading order, and there are multiple ways to interact with it from a keyboard, touch input, and voice. But if the reading system or ebook reading software does not have an available screen reader or text-to-speech voice output capabilities, then that ebook is still not accessible to a student who is blind. If there is no microphone in the device, then voice input is unavailable for students who may not have use of their hands to perform required tasks.

 

Opportunities

Fortunately, there are increasing opportunities for producers of multimedia interactives to make them accessible. PhET is working to make their simulations accessible by adding keyboard navigation, audio descriptions, and sonification (the use of non-speech sounds to convey information). PhET has resolved some of the accessibility challenges they identified in their simulations. For instance, the PhET interactive simulation about static electricity has been made accessible to visually impaired students in that they can manipulate the arm and leg using the arrow keys and the electrical charge can be heard auditorily (Smith et al., 2017).

More guidelines are becoming available for developers to create accessible interactives. Technically minded developers can read a general primer on accessible interactive objects, such as scientific graphics or laboratory simulations, in the report entitled “Accessible Dynamic Scientific Graphics” sponsored by the DIAGRAM Center. The online report includes a demonstration of how to make dynamic scientific images accessible to users who are blind or visually impaired. “Interactive Scientific Graphics: Recommended Practices for Verbal Descriptions” is a detailed report that describes the best practices for making interactive scientific graphics pedagogically and functionally equivalent for users who are blind. Publishers of ebooks that feature interactivity can also take advantage of the new Global Certified Accessible program that assesses digital books in the EPUB format for their accessibility and certifies those that meet the requirements.

Mobile apps, new VR and AR experiences, and digital learning platforms like Blackboard are increasingly aiming for more inclusion. For instance, the DIAGRAM Center is working with WGBH NCAM on making accessible multimedia for young learners based on the popular science program PEEP and the Big Wide World. The conversation from the gaming perspective is being led by groups like AbleGamers, which is dedicated to making it possible for people to play video games no matter their disability. Devices that allow third-party Bluetooth or Wi-Fi enabled assistive technology to connect with them now offer new ways to control the device beyond what was built in, such as external ergonomic keyboards, joysticks, puff switches, and braille input and/or output devices that can connect and control the device and ultimately the multimedia interactive.

Augmented environments can help students learn in new and enriched ways. For persons with physical impairments or those in rehabilitation services, the educational benefits could be far-reaching. A study that was conducted in 2010 (Burke et al.), found that the use of real objects interacting with computer-enhanced environments greatly improved the rehabilitation of stroke victims. In another study, researchers gave oral and visual cues to students with disabilities through the use of augmented reality to gauge their ability to make decisions. They found that students were very motivated to use this technology, especially those students on the autism spectrum (Richard et al., 2007).

In a study conducted in 2001, researchers found that the use of virtual reality could help in the identification of Attention Deficit Hyperactivity Disorder (ADHD) (Rizzo et al., 2001). They found that the use of virtual reality (VR) could improve assessment by providing more consistent stimuli and measurement. “VR could offer potential for cognitive assessment and rehabilitation within simulated ‘real-world’ functional testing and training environments with an aim towards improving ecological validity” (Rizzo et al., 2001, p.31).

At the University of Central Florida, Dr. Lisa Dieker and her team have created TeachLivE, a virtual classroom designed to improve teacher practice and student learning. Using avatars, teachers-in-training have the opportunity to practice classroom management, student interaction, and curriculum delivery before actually stepping into a live classroom. The avatars are responsive and give real-time feedback to the teachers. One of the avatars is on the autism spectrum, and the teachers have an opportunity to interact with him to better understand his needs before working with “real” students.

 

Stories from the Field

 

SAS enables visually impaired to ‘visualize’ data

Representing numerical data in multiple formats that is accessible has been a challenge. Charts, graphs, and maps are usually just a static visual representation of the underlying data with no way to interact with the data. By making these interactive, it allows students to explore these images and get at the data in alternative ways.
“The inaccessibility of online graphs is a huge barrier for students who are blind or visually impaired,” says Sarah McManus, Director of Digital Learning for the Deaf and Blind at the North Carolina Department of Public Instruction. “The ability to interact with and understand data has the potential to increase these students’ problem-solving skills exponentially and open up new opportunities, especially in STEM education.”

SAS is a company devoted to making data visualizations accessible. The SAS Graphics Accelerator dynamically generates alternative representations for data, including verbal descriptions that explain the data in a human friendly way, as well as sonification. SAS provides a number of accessible examples of interactive accessible charts, tables, and maps (SAS, n.d.).

“Reach for the Stars, Touch, Look, Listen, Learn” – a collaboration between the Space Telescope Science Institute and SAS – is an example of an interactive iBook for Apple’s iPad that immerses the student in an accessible exploration of astronomy. This multimedia, interactive iBook utilizes the accessibility features of the iPad, videos from famous scientists with closed captioning, human narration, interactive quizzes, the ability to create custom notes, sonification to represent data using sound, and 3D models that can be downloaded and printed to bring the Hubble telescope to life.

 

Conclusions / Actions

Multimedia interactives help students learn by engaging multiple senses. Providing more than one media experience for a particular interactive can benefit students with disabilities. However, a multimedia interactive may not be fully accessible to all students. The material will need to be checked for accessibility (i.e., are images and videos described, is there a transcript for the audio, and so on). Even if the content has accessibility features, the student’s hardware device must also be considered (i.e., is the content readable by assistive technology such as screen readers, external keyboards, braille displays, or switches).

 

Educators

1. When searching for multimedia interactives, make sure that the interactives include accessible images, videos, audio, and simulations (i.e., images are described, videos are captioned, audio has transcripts, and simulations are operable by those with physical impairments).
2. When selecting an EPUB ebook, check to see whether it has been certified as being accessible. For more information on certification, follow the efforts of the Global Certified Accessible.

 

Parents

1. Schedule a meeting with your child’s teacher prior to the start of school and find out what forms of multimedia interactives he or she is planning to use in the classroom. Check to see if these will meet your child’s needs, and if possible, have them remediated by adding in any missing content.
2. Find out what reading systems, computers, or tablets will be used in the classroom and check to see what assistive technology could be added to those systems to support your child’s needs.
3. Be proactive, and research what reading systems for EPUBs are best for your child and see if this is something your child can use in the classroom. See, for example, EPUB Test from the Book Industry Study Group (BISG).

 

Students

1. Experiment with different reading systems to find the one that is best for you. Make sure you know how to turn on captioning in videos when present, or how to have an image description read to you.
2. Experiment with some of the PhET simulations to understand what is currently possible in accessible simulations.

 

References

• R. (n.d.). A brief guide to interactive multimedia and the study of the United States. Retrieved July 24. 2017 from the World Wide Web: http://www.georgetown.edu/faCUI1Y/bassr/mullimedia.html
• Bellini, H., Chen, W., Sugiyama, M, Shin, M., Alam, S., Takayama, D. (2016). Virtual & Augmented Reality: Understanding the Race for the Next Computing. Published January 13, 2016. PDF Retrieved August 23 http://www.goldmansachs.com/our-thinking/pages/technology-driving-innovation-folder/virtual-and-augmented-reality/report.pdf
• Burke, J. W., McNeill, M. D. J., Charles, D. K., Morrow, P. J., Crosbie, J. H., & McDonough, S. M. (2010, March). Augmented reality games for upper-limb stroke rehabilitation. In Games and Virtual Worlds for Serious Applications (VS-GAMES), 2010 Second International Conference on Games and Virtual Worlds for Serious Applications, Braga, 2010, pp. 75-78.
• Cuendet, S., Bonnard, Q., Do-Lenh, S., & Dillenbourg, P. (2013). Designing augmented reality for the classroom. Computers & Education, 68, 557-569.
• (n.d.). From Written to Digital: The New Literacy. Retrieved August 23, 2017, from http://er.educause.edu:81/articles/2016/3/from-written-to-digital-the-new-literacy
• Encyclopaedia Britannica. (n.d.). Interactive multimedia. Retrieved August 16, 2017, from https://www.britannica.com/technology/interactive-multimedia
• Graham, L. (2017, February 15). Digital games market to see sales worth $100 billion this year: Research. Retrieved August 7, 2017, from https://www.cnbc.com/2017/02/15/digital-games-market-to-see-sales-worth-100-billion-this-year-research.html
• Lopez, N. (2015, March 16). YouTube’s Mobile-Friendly Cards Will Replace Annotations. Retrieved August 23, 2017, from https://thenextweb.com/insider/2015/03/16/youtube-introduces-interactive-cards-to-complement-annotations-and-they-work-on-mobile-too/ – .tnw_acNVudWf
• Perez, S. (2016, September 08). Pokémon Go becomes the fastest game to ever hit $500 million in revenue. Retrieved August 18, 2017, from https://techcrunch.com/2016/09/08/pokemon-go-becomes-the-fastest-game-to-ever-hit-500-million-in-revenue/
• Richard, E., Billaudeau, V., Richard, P., & Gaudin, G. (2007, September). Augmented reality for rehabilitation of cognitive disabled children: A preliminary study. In Virtual Rehabilitation, 2007(pp. 102-108). IEEE.
• Rizzo, A. A., Buckwalter, J. G., Bowerly, T., Van Der Zaag, C., Humphrey, L., Neumann, U., & Sisemore, D. (2000). The virtual classroom: a virtual reality environment for the assessment and rehabilitation of attention deficits. CyberPsychology & Behavior, 3(3), 483-499.
• (n.d.). SAS enables visually impaired to ‘visualize’ data. Retrieved July 23, 2017, from https://www.sas.com/en_us/news/press-releases/2017/february/sas-graphics-accelerator-education.html
• Smith, T. L., Lewis, C., & Moore, E. (2017, March). Description strategies to make an interactive science simulation accessible. Lecture presented at CSUN Assistive Technology Conference, San Diego, Ca.
• Terdiman, D. (2017, June 07). The 10 Most Innovative Companies In VR/AR 2017. Retrieved August 16, 2017, from https://www.fastcompany.com/3068430/the-10-most-innovative-companies-in-vr-ar-2017
• Vanian, J. (2016, March 22). 8 Crazy Examples Of What’s Possible In Virtual Reality. Retrieved August 16, 2017, from http://fortune.com/2016/03/22/sony-playstation-vr-virtual-reality/

 

Resources

• CSUN Assistive Technology Conference
• Standardizing Textual Descriptions of Interactive Graphics – Kyle Keane, PhD, Wolfram Research
• New Models for Creating Accessible Interactive Widgets for STEM Learning – Kyle Keane, PhD , Madeline Rothberg, NCAM – DIAGRAM Center Webinar
• A Brief Guide to Interactive Multimedia and the Study of the United States
• 12 Principles of Multimedia Learning – Richard E. Mayer Cambridge Press, 2001
• Principles for multimedia learning with Richard E. Mayer
• Benefits and Pitfalls of Multimedia and Interactive Features in Technology-Enhanced Storybooks
• 20 FREE TOOLS FOR CREATING MULTIMEDIA AND INTERACTIVES
• RIMES – Embedding Interactive Multimedia Exercises in Lecture Videos
• Producing interactive multimedia learning resources
• Interactive Digital Learning Materials for
• Kindergarten Students in Bangladesh
• Moving towards multimedia learning
• Think You’re An Auditory Or Visual Learner? Scientists Say It’s Unlikely
• Benefits and Pitfalls of Multimedia and Interactive Features in Technology-Enhanced Storybooks
• Web Design: Video Captioning, Audio Descriptions, and Accessible Media Players
• Producing Accessible Multimedia
• W3C Multimedia Accessibility FAQ
• National Center for Accessible Media (NCAM) – The National Center for Accessible Media is a growing collection of resources for developers and users interested in ways to make rich media accessible to people with disabilities.
• MAGpie is a free tool available through NCAM to help in the creation of captions and audio descriptions for multimedia.
• YouTube Captioning – An article from YouTube discussing how to add captions and transcripts to your videos on their site.
• TopHat 2016’s Biggest Stories in Ed Tech – A blog post which featured “Minecraft blows open educational gaming”
• SAS – Graphics Accelerator transforms graphs and charts into sound to paint audio picture

Published: 2017-08-31