Wenhao Ran received his PhD in the Institute of Semiconductors, Chinese Academy of Sciences in 2022. He is currently a postdoctoral researcher at the School of Integrated Circuits and Electronics, Beijing Institute of Technology, Beijing. His current research focuses on flexible biomimic vision system with in-memory sensing and computing.
Zhuoran Wang received his PhD in the department of Mining and Materials Engineering from the McGill University, QC, Canada in 2017. In 2019 he joined the Institute of Photonic Sciences (ICFO), Barcelona, as a postdoctoral /Marie-Curie research fellow. He is currently a professor at the School of Integrated Circuits and Electronics, Beijing Institute of Technology, Beijing. His current research focuses on flexible and infrared optoelectronic sensors for biomimic vision.
Guozhen Shen received his PhD degree in Chemistry from the University of Science and Technology of China. He is currently a professor at the School of Integrated Circuits and Electronics, Beijing Institute of Technology (BIT), and the director of the Institute of Flexible Electronics and Intelligent Manufacturing. Before joining BIT, he worked at Hanyang University (Korea), National Institute for Materials Science (Japan), University of Southern California (US), and Huazhong University of Science and Technology (China), the Institute of Semiconductors, CAS (China). His current research focuses on flexible electronic devices for artificial intelligence and healthcare monitoring.With the rapid development of drones and autonomous vehicles, miniaturized and lightweight vision sensors that can track targets are of great interests. Limited by the flat structure, conventional image sensors apply a large number of lenses to achieve corresponding functions, increasing the overall volume and weight of the system. As one of the highly sophisticated and remarkably organs, the biological visual systems have been optimized through evolution and have the advantages of broad field of view (FoV), profound depth of field, high light sensitivity, low off-axis aberrations and miniaturization[1−3]. The bionic visual system inspired by them shows fascinating performances and potential for application in static and dynamic imaging functionalities[4].
As a typical representative of biological visual systems, the unique structure of the fovea in avian eyes can magnify distant targets by refracting incident light. The ultraviolet and RGB light perception capabilities brought by four corresponding cones enrich the information of the target objects and effectively distinguish them from the background. The uneven distribution of cones reduces the surrounding information while increasing the resolution of the target object. Therefore, the motion sensitivity of the target object is higher. It has a natural advantage in detecting and tracking distant moving targets in dynamic environments. Inspired by avian eyes, Kim et al. realize a bionic visual system with fovea and multispectral imaging by combining designed an artificial fovea and vertically stacking four perovskites photodetector arrays with different bandgaps[5] (Sci Robot, 2024, https://doi.org/10.1126/scirobotics.adk6903). This system enables colored object recognition without additional color filters and distant target object magnification. The YOLO v5 algorithm is used to simulate the tracking and recognition of moving target objects, which confirms the application potential of the system in the field of target tracking.
Specifically, this novel bionic vision sensor is of three features: artificial fovea, vertically stacked multispectral perovskite photodetectors, and pixel density variation. The artificial fovea imitates the fovea of a falcon (Fig. 1(a)) and is prepared by 3D printing and PDMS transfer. Its shape adopts a Gaussian profile, the size of which is optimized assisted by optical simulation to ensure the appropriate magnification level (Fig. 1(b)). This design ensures uninterrupted magnification without blind spots or image overlap. In addition, the difference in magnification between the fovea and the peripheral regions can magnify and focus the object in the fovea while images are blurred in the peripheral regions, thus facilitate the detecting of the moving objects (Fig. 1(c)).
DownLoad:
The vertically stacked photodetectors are utilized to achieve tetrachromatic imaging without color filters. Perovskite materials are herein applied as their bandgaps can be easily tuned by adjusting the compositions, by the stacking of which via a transfer printing process, UV−B−G−R colored vision are realized without additional filters. More importantly, this vertical stacking strategy is beneficial in suppressing the optical artifacts caused by color image reconstruction, and more importantly, overcoming the pixel density limitation for the in-plane color sensing arrays. Fig. 1(d) shows the cross-sectional view of the multispectral image sensor. The test results shown in Fig. 1(e) show that each photodetector is responsive solely to the target wavelength. Figs. 1(f) and 1(g) show the multispectral imaging capabilities tested with a color checker. The results further demonstrate that the multispectral image sensor can successfully extract R, G, B, and UV information without filters and can merge multiple colors to reconstruct a color image.
Moreover, the avian’s eyes mimetic unique design of uneven pixel density ensures a high pixel density in the fovea region (Fig. 1(h)), achieving high resolution of the target spotting, and simultaneously ensuring a wide field of view in the peripheral regions (Fig. 1(i)), which is conducive to the effective allocation of computing resources during image processing. Finally, the YOLO v5 algorithm is used to simulate the ability of vision system to detect object motion, revealing its potential in tracking moving objects. The results illustrate that compared with the original image, the fovea module achieves higher confidence score by magnifying the target object and enhanced object recognition efficiency (Figs. 1(j)−1(l)). The wider magnitude of motion indicates that the foveal module is more proficient at detecting the motion of objects and enhances capability for motion detection by more than 3.6-fold.
In summary, this avian vision inspired bionic vision sensor enabling magnification of target objects through artificial fovea design, featuring vertically stacked perovskite photodetectors of four different bandgaps with filterless tetrachromatic imaging capacity, uneven pixel distribution density caused different spatial resolutions in the fovea and peripheral regions, which helps to effectively allocate computing resources during image processing while ensuring high resolution of the target object in the fovea and wide field of view in the peripheral regions. In addition to previously reported bionic image sensors inspired by the eyes of humans[6], aquatic creatures[7], insects[8], etc., featuring large FoV, low aberrations, amphibious vision, large LDR, etc., this hawk-eye image sensor compliment this family and provides new insights for developing miniaturized telescope vision containing rich spectral and dynamic information. However, further increasing pixel density to a complementary metal oxide semiconductor (CMOS) comparable level and reducing the complexity of the stacking procedure remain challenging, where new strategies in optoelectronic monolithic and heterogeneous integration in a flexible platform are desired[9−11].
DownLoad:
DownLoad:
