New Smart Cameras Help People with Vision Disability
Author: Penn State
Published: 2015/02/05 - Updated: 2021/09/08
Topic: Disability Visual Aids (Publications Database)
Page Content: Synopsis Introduction Main Item
Synopsis: Visual Cortex on Silicon aims to create smart cameras that will replicate or even surpass abilities of human vision system.
• To help make shopping easier for people with vision disabilities Jack Carroll has completed a study that explored how smart cameras could eventually guide visually impaired shoppers to find the items they need.
• The project - dubbed Visual Cortex on Silicon - is led by Penn State Computer Science and Engineering Distinguished Professor Vijaykrishnan Narayanan, and Penn State is the lead university alongside seven other universities.
Introduction
Standing in the grocery store, you scan the peanut butter jars looking for the one on your list. Your eyes flit from label to label until they land on the familiar red, blue and green jar, and you reach to pick it up and place it in your cart. Then, it's on to the next item on your list.
Main Item
Picking something up at the grocery store seems simple, but it's a task that relies heavily on your sense of sight. You need to find the right item, pick it up and then place it safely in your basket. But for people with vision disabilities, it's a task that's often difficult or impossible for them to do on their own.
To help make shopping easier for people with vision disabilities, Jack Carroll - a distinguished professor in the College of Information Sciences and Technology (IST) at Penn State - has completed a study that explored how smart cameras could eventually guide visually impaired shoppers to find the items they need. Carroll worked on the study with Mary Beth Rosson, interim dean of the College of IST, as well as graduate student Jake Weidman as part of a $10 million National Science Foundation-funded project that's seeking to replicate the human vision system using information technology (IT).
"This is one of the biggest computing projects Penn State's been involved with," said Carroll. "I've been working with what we do with the technology after the camera is available, specifically in a shopping scenario. Shopping is so universal, and it signifies normalcy - it's important that people with vision disabilities be able to do it, too."
The project - dubbed Visual Cortex on Silicon - is led by Penn State Computer Science and Engineering Distinguished Professor Vijaykrishnan Narayanan, and Penn State is the lead university alongside seven other universities.
"This project brings together the strengths and efforts of technical leaders in multiple disciplines," said Narayanan. "It draws on the expertise of several industry and national lab partners. A major goal is advancing the technologies available for people with vision disabilities."
Visual Cortex on Silicon aims to create sophisticated "smart" cameras that will replicate or even surpass the abilities of the human vision system, which can interpret complex scenes and complete complicated tasks while using less than 20 watts of power. The cameras will be used in mobile devices, such as smartphones, so energy efficiency is key to fitting a complex system into a small piece of hardware.
Researchers - who are each tackling individual pieces of the larger project - hope the cameras will not just be able to record images but also interpret them. For example, the cameras will not just recognize an item as a jar of peanut butter but also be able to determine whether it's the specific kind the user needs - a scenario that was the focus of Carroll's study.
Carroll and his team mocked up a grocery aisle in his lab, complete with food and other items filling the shelves.
He wanted to see how participants with vision disabilities would interact with the camera, which would scan the items on the shelves and alert the participant when the item they were searching for was near.
But, since the new cameras haven't been finished yet, an actual prototype wasn't available to test. So, Carroll and his team used the "Wizard of Oz" technique, in which the participants interact with an apparatus that's being controlled by an out-of-sight person.
"In the movie, the wizard was actually an illusion controlled by a man behind the curtain. These types of experiments are similar," said Carroll. "For this one, the cameras aren't ready so we had to simulate how they might work in the future. It allows us to see how participants would interact with the camera and tweak any problems that arise. We don't want to wait until the cameras are finished being developed and then start from scratch on the apparatus."
Participants strapped iPods on their foreheads that would send a video feed of the shelf to a graduate student who would then list the items currently in view. This simulates what they would eventually be able to do with one of the new cameras.
Once the participant knew they were standing in front of the item they needed, the next step was guiding them to pick it off the shelf. This proved a bit more difficult than the first step.
"At first we tried using verbal cues to help them locate it - 'higher' or 'lower,' for example, as the participants moved their hands," says Carroll. "This worked well, but we didn't want to stop there and are also looking into other guiding methods."
Carroll is now investigating a method using haptic sensors, which give tactile feedback through vibration.
The "buzzes" would become stronger as the participants' hands got closer to their desired items. Carroll says he likes this method because a lot of speech is needed for even small directions and other people's speech can interfere with feedback.
"Jack's work is key because it enables the new cameras being researched and built in this project to meet expectations and understand what kind of assistance visually impaired people will need from these new technologies," Narayanan said.
Shortly after Carroll's study was completed, the Visual Cortex on Silicon team was able to use research completed by other team members on hardware design and vision algorithms to build an actual prototype of the apparatus. The team demonstrated the system at the project's annual review.
Narayanan says the Visual Cortex on Silicon project - which is funded through September 2018 - came about after a previous effort designing cameras.
"In a prior project, we used brain-inspired algorithms and configurable architectures to detect and track objects for aerial cameras," Narayanan said. "The system was more energy efficient compared to other current approaches, and it made us want to work toward cameras that are applicable across a wider spectrum of vision-related tasks."
Attribution/Source(s):
This quality-reviewed publication was selected for publishing by the editors of Disabled World due to its significant relevance to the disability community. Originally authored by Penn State, and published on 2015/02/05 (Edit Update: 2021/09/08), the content may have been edited for style, clarity, or brevity. For further details or clarifications, Penn State can be contacted at cse.psu.edu. NOTE: Disabled World does not provide any warranties or endorsements related to this article.
1 - Revolutionary Vision Therapy: AmblyoPlay Unveils Peripheral Sight Enhancer - AmblyoPlay launches Active Peripheral Stimulation (APS) intended to advance treatment of peripheral vision deficiencies and improve effectiveness of home-based vision therapy.
2 - WorldScribe: Real-Time Surrounding Descriptions for the Visually Impaired - A world of color and texture could soon become more accessible to people who are blind or have low vision through new software called WorldScribe.
3 - Tactile Lithophane Makes Scientific Data Accessible to Students With Blindness - Tactile learning device makes science accessible to students with blindness or low vision (BLV) opens possibility of transfer of scientific data or images for sighted students into functional formats for students with blindness.
Page Information, Citing and Disclaimer
Disabled World is a comprehensive online resource that provides information and news related to disabilities, assistive technologies, and accessibility issues. Founded in 2004 our website covers a wide range of topics, including disability rights, healthcare, education, employment, and independent living, with the goal of supporting the disability community and their families.
Cite This Page (APA): Penn State. (2015, February 5 - Last revised: 2021, September 8). New Smart Cameras Help People with Vision Disability. Disabled World. Retrieved December 12, 2024 from www.disabled-world.com/assistivedevices/visual/cameras.php
Permalink: <a href="https://www.disabled-world.com/assistivedevices/visual/cameras.php">New Smart Cameras Help People with Vision Disability</a>: Visual Cortex on Silicon aims to create smart cameras that will replicate or even surpass abilities of human vision system.
While we strive to provide accurate and up-to-date information, it's important to note that our content is for general informational purposes only. We always recommend consulting qualified healthcare professionals for personalized medical advice. Any 3rd party offering or advertising does not constitute an endorsement.