Seeing the Unseen, Panasonic Develops the Worlds Highest Sensitivity Hyperspectral Imaging Technology, Opening Up Industrial and Consumer Applications

Imagine using an everyday camera to instantly detect fruit that isn’t sweet, medication that may be impure or coatings that don’t meet quality standards. These and other useful applications are possible through hyperspectral sensing, a form of imaging that goes beyond human vision. 

Panasonic has developed the world’s highest sensitivity hyperspectral imaging technology while using a conventional camera with a new filter at standard video rates, allowing this advanced image processing technique to open up many new applications in machine vision. 1  

The human eye mostly relies on the red, green and blue wavelengths to gather visual information from a scene. Hyperspectral imaging sensors are able to divide the color spectrum into dozens or hundreds of wavelength bands and identify the spectrum for each pixel in an image. 2  

As a result, this approach gleans much more information from a scene. Another important aspect is that some objects can leave a telltale “fingerprint” of their presence in the electromagnetic spectrum. That’s why hyperspectral imaging can be used to look for things in an image, for instance the spectral fingerprint of oil in minerology imaging. 

One drawback to wider use of hyperspectral imaging in industrial and, eventually, consumer applications, however, is that it has relied on optical elements such as prisms and filters to pass light of specific wavelengths. This physical limit creates image limitations. The result is low sensitivity, resolution or frame rate compared with conventional monochrome/color cameras. If a scene isn’t brightly illuminated, like an LED-lit office, hyperspectral imaging can’t be used. 

Researcher Motoki Yako runs a test on the Panasonic hyperspectral imaging system, which introduces a new filter that mixes the colors together and then separates them to increase sensitivity over 10 times compared to conventional systems

To overcome this problem, Panasonic’s researchers turned to a signal processing technique known as compressed sensing. It allows for images to be reconstructed from under-sampled signals. Compressed sensing has been used to peer into our bodies using magnetic resonance imaging (MRI) as well as black holes in deep space. The researchers managed to develop a hyperspectral imaging system that is sensitive enough to be used in a darkened room, and a frame rate and resolution compatible with conventional cameras including video applications. 

“ We managed to increase sensitivity over 10 times compared to conventional systems, ” says Motoki Yako, staff researcher at Panasonic’s Applied Materials Technology Center. “This ‘bright hyperspectral camera’ is a world first and delivers a technology with high usability. The key innovation is the design of a new filter that mixes the colors together and then separates them.” 

A study on the research results, coauthored with colleagues from imec, a Belgian research institute, was published in the peer-reviewed journal Nature Photonics. 3 The researchers wrote that the “ camera holds great promise for the adoption of [hyperspectral] technology in real-world scenarios, including consumer applications such as smartphones and drones. ” 

To illustrate how the camera has practical video uses, the study features a hyperspectral video with a VGA resolution of 32 frames per second showing a metronome ticking away in everyday office lighting, a scene that would be dark if imaged by a conventional hyperspectral camera. Half the frame is a standard RGB camera and half is the hyperspectral camera, showing their similar resolution and sensitivity.

The image sensor detects light and a special filter does the thinning out, which improves sensitivity since multiple wavelengths of light are transmitted and detected to transmit about 45% of the incident light, resulting in the world’s highest sensitivity of hyperspectral imaging

The camera is the result of the researchers’ eagerness to commercialize cutting-edge signal processing and their ability to work flexibly. Yako joined Panasonic in 2019 after earning a PhD in photonics, a branch of physics concerned with photons and their applications. He specialized in silicon photonics, a discipline that was finding industrial applications, and decided to bring his talents to Panasonic because he wanted to introduce new things to the world. 

It was there that he met project leader Atsushi Ishikawa, another young scientist who had done postgraduate research in spectroscopy before joining the company in 2018. Ishikawa also wanted to contribute to society by doing basic research that leads to products. 

In 2019, Ishikawa and Yako began their work on a more sensitive, practical hyperspectral camera. The challenge involved fields such as manufacturing and software that were outside their areas of expertise. In an unusual process for Japanese companies, project members were flexibly assigned, and other experts joined the team, made contributions, and then left. The scientists credit this constantly evolving, multidisciplinary nature of the group with the success of the project.     

“ Panasonic’s unique strength lies in the fact that experts could work closely together in the three key fields needed to bring about this innovation, ” says Ishikawa, a leader of the team. “Namely, the technology for hyperspectral imaging, manufacturing innovation for the filter system, and various other departments that worked on software development.” 

Practical applications of the hyperspectral imaging technology, which works in low light, include the detection of the ripeness of fruits and vegetables, contamination levels of soil, surveillance of marine plastics and, if miniaturized for smartphones, an as yet unimagined variety of consumer uses

The researchers are now working with Panasonic manufacturing engineers who are carrying out image inspection experiments with a prototype version of the device. The first likely commercial applications will include factory automation sensing for product inspection. 4 For instance, the sugar content of tomatoes can affect their spectral fingerprint, revealing if they are sweet or not. Since it works in relatively low light, the camera does not need a lighting system that would expose perishable food to higher temperatures. 

More rugged versions of the system could be mounted on drones or Internet of Things (IoT) devices for surveying marine plastics, soil contamination and other environmental uses. If miniaturized, the camera could be installed on smartphones for consumer applications that we haven’t even dreamed of yet.    

“ I’m eager to see the technologies we have developed being released in products, ” says Ishikawa. “Compared to conventional spectroscopy and hyperspectral technology used in labs, our technology works in low light and is easy to use. It could be used as an imaging technology with a large amount of color information on a daily basis. By that time, AI technology will have become widespread, and it would be good if it becomes a technology that improves people’s lives.”