J. Semicond. >  Just Accepted

ARTICLES

A 2D/3D Vision Chip based on Organic Substrate 3D Package

Siyuan Wei1, 2, §, Quanmin Chen3, §, Jingyi Yu1, 3, §, Xuanzhe Xu1, 2, Yuxiao Wen1, 2, Runjiang Dou1, 2, Shuangming Yu1, 2, Guike Li1, 2, Kaiming Nie3, Jie Cheng5, Jiangtao Xu3, , Liyuan Liu1, 4, and Nanjian Wu1, 2

+ Author Affiliations

 Corresponding author: Jiangtao Xu, xujiangtao@tju.edu.cn; Liyuan Liu, liuly@semi.ac.cn

DOI: 10.1088/1674-4926/25010030CSTR: 32376.14.1674-4926.25010030

PDF

Turn off MathJax

Abstract: This paper describes a 2D/3D vision chip with integrated sensing and processing capabilities. The 2D/3D vision chip architecture includes a 2D/3D image sensor and a programmable visual processor. In this architecture, we design a novel on-chip processing flow with die-to-die image transmission and low-latency fixed-point image processing. The vision chip achieves real-time end-to-end processing of convolutional neural networks (CNNs) and conventional image processing algorithms. Furthermore, an end-to-end 2D/3D vision system is built to exhibit the capacity of the vision chip. The vision system achieves real-timing applications under 2D and 3D scenes, such as human face detection (processing delay 10.2 ms) and depth map reconstruction (processing delay 4.1 ms). The frame rate of image acquisition, image process, and result display is larger than 30 fps.

Key words: vision chip, 2-D/3-D image processing, near-sensor computing, convolutional neural networks



[1]
Ishikawa M, Ogawa K, Komuro T, et al. A CMOS vision chip with SIMD processing element array for 1 ms image processing. 1999 IEEE International Solid-State Circuits Conference. Digest of Technical Papers (ISSCC), 1999, 206
[2]
Yamazaki T, Katayama H, Uehara S, et al. 4.9 A 1ms high-speed vision chip with 3D-stacked 140 GOPS column-parallel PEs for spatio-temporal image processing. 2017 IEEE International Solid-State Circuits Conference (ISSCC), 2017, 82
[3]
Kumagai O, Ohmachi J, Matsumura M, et al. 7.3 A 189 × 600 back-illuminated stacked SPAD direct time-of-flight depth sensor for automotive LiDAR systems. 2021 IEEE International Solid-State Circuits Conference (ISSCC), 2021, 110
[4]
Withanage K I, Lee I, Brinkworth R, et al. Fall recovery subactivity recognition with RGB-D cameras. IEEE Trans Ind Inform, 2016, 12(6), 2312 doi: 10.1109/TII.2016.2590339
[5]
Kim G, Lee K, Kim Y, et al. A 1.22 TOPS and 1.52 mW/MHz augmented reality multicore processor with neural network NoC for HMD applications. IEEE J Solid State Circuits, 2015, 50(1), 113 doi: 10.1109/JSSC.2014.2352303
[6]
Lau J H. Recent advances and trends in advanced packaging. IEEE Trans Compon Packag Manuf Technol, 2022, 12(2), 228 doi: 10.1109/TCPMT.2022.3144461
[7]
Sharda J, Manley M, Kaul A, et al. Design and thermal analysis of 2.5D and 3-D integrated system of a CMOS image sensor and a sparsity-aware accelerator for autonomous driving. IEEE J Electron Devices Soc, 2024, 12, 426 doi: 10.1109/JEDS.2024.3354621
[8]
Chen Q M, Nie K M, Gao J, et al. A 1920 × 1080 array 2-D/3-D image sensor with 3-μs row-time single-slope ADC and 100-MHz demodulated PPD locked-In pixel. IEEE J Solid State Circuits, 2024, PP(99), 1
[9]
Wei S Y, Ning K, Kang L, et al. A real-time 2D/3D perception visual vector processor for 1920 × 1080 high-resolution high-speed intelligent vision chips. IEEE Trans Circuits Syst I Regul Pap, 2024, 71(2), 740 doi: 10.1109/TCSI.2023.3339784
[10]
Xu M M, Zhao M X, Yu S M, et al. Rounding shift channel post-training quantization using layer search. 2021 4th International Conference on Artificial Intelligence and Big Data (ICAIBD), 2021, 545
[11]
Zhao M X, Peng J B, Yu S M, et al. Exploring structural sparsity in CNN via selective penalty. IEEE Trans Circuits Syst Video Technol, 2022, 32(3), 1658 doi: 10.1109/TCSVT.2021.3071532
[12]
Zheng X M, Zhao M X, Luo Q, et al. A chip-level verification method for programmable vision chip based on deep learning algorithms. 2020 IEEE 5th International Conference on Integrated Circuits and Microsystems (ICICM), 2020, 281
[13]
Kim D, Lee S, Park D, et al. Indirect time-of-flight CMOS image sensor with on-chip background light cancelling and pseudo-four-tap/two-tap hybrid imaging for motion artifact suppression. IEEE J Solid State Circuits, 2020, 55(11), 2849 doi: 10.1109/JSSC.2020.3021246
[14]
Eki R, Yamada S, Ozawa H, et al. 9.6 A 1/2.3 inch 12.3 Mpixel with on-chip 4.97 TOPS/W CNN processor back-illuminated stacked CMOS image sensor. 2021 IEEE International Solid- State Circuits Conference (ISSCC), 2021, 154
[15]
Lepecq M, Dalgaty T, Fabre W, et al. End-to-end implementation of a convolutional neural network on a 3D-integrated image sensor with macropixel array. Sensors, 2023, 23(4), 1909 doi: 10.3390/s23041909
[16]
Sharda J, Li W T, Wu Q C, et al. Temporal frame filtering for autonomous driving using 3D-stacked global shutter CIS with IWO buffer memory and near-pixel compute. IEEE Trans Circuits Syst I Regul Pap, 2023, 70(5), 2074 doi: 10.1109/TCSI.2023.3252486
Fig. 1.  (Color online) 2D/3D vision chip architecture[8, 9].

DownLoad: Full-Size Img PowerPoint
Fig. 2.  (Color online) Architecture of the 2D/3D vision system.

DownLoad: Full-Size Img PowerPoint
Fig. 3.  (Color online) 2D/3D vision system workflow.

DownLoad: Full-Size Img PowerPoint
Fig. 4.  (Color online) Three-view drawing of the 2D/3D vision chip.

DownLoad: Full-Size Img PowerPoint
Fig. 5.  (Color online) NN quantization and mapping process.

DownLoad: Full-Size Img PowerPoint
Fig. 6.  (Color online) The vision chip’s fixed-point inference flow of NNs

DownLoad: Full-Size Img PowerPoint
Fig. 7.  (Color online) The 2D/3D vision system demo.

DownLoad: Full-Size Img PowerPoint
Fig. 8.  (Color online) Timing chart of on-chip processing. (a) Human face detection. (b) Depth map reconstruction.

DownLoad: Full-Size Img PowerPoint
Fig. 9.  (Color online) 2D/3D application demo. (a) 2D Human face detection. (b) 3D Depth map reconstruction.

DownLoad: Full-Size Img PowerPoint
Fig. 10.  (Color online) Heat map of the vision chip in 2D mode.

DownLoad: Full-Size Img PowerPoint

Table 1.   Comparison with other NN vision chip.

Ref. [14]Ref. [15]Ref. [16]This work
Process (nm) Sensor/Processor65/2213040/22110/55
Max Clock frequency (MHz)262.580100200
Processing Data type (int)8b, 16b, 32b4b8b8b, 16b, 32b (activation)
Imager resolution (H/V)4056 × 30401024 × 7681296 × 7201920 × 1080
Peak Performance (8b)1.21 TOPS61 GOPS9.05 TOPS409.6 GOPS
Energy efficiency (OPS/W)1.89 (CNN)No mentioned651.49 (CNN)
Processing latency (ms)4.5−4.8
(end-to-end CNN)		3.8
(end-to-end simple CNN)		1.15
(need post process)		10.3
(end-to-end CNN)
Algorithms SupportCNN3-layer CNNTemporal frame filter2D&3D CV/CNN
Imager data modalRGBGrayRGBiTOF/Gray
Frame rate120 (2028 × 1520 2D)340 (2D)30 (2D)300 (2D)/30 (3D)
DownLoad: CSV
[1]
Ishikawa M, Ogawa K, Komuro T, et al. A CMOS vision chip with SIMD processing element array for 1 ms image processing. 1999 IEEE International Solid-State Circuits Conference. Digest of Technical Papers (ISSCC), 1999, 206
[2]
Yamazaki T, Katayama H, Uehara S, et al. 4.9 A 1ms high-speed vision chip with 3D-stacked 140 GOPS column-parallel PEs for spatio-temporal image processing. 2017 IEEE International Solid-State Circuits Conference (ISSCC), 2017, 82
[3]
Kumagai O, Ohmachi J, Matsumura M, et al. 7.3 A 189 × 600 back-illuminated stacked SPAD direct time-of-flight depth sensor for automotive LiDAR systems. 2021 IEEE International Solid-State Circuits Conference (ISSCC), 2021, 110
[4]
Withanage K I, Lee I, Brinkworth R, et al. Fall recovery subactivity recognition with RGB-D cameras. IEEE Trans Ind Inform, 2016, 12(6), 2312 doi: 10.1109/TII.2016.2590339
[5]
Kim G, Lee K, Kim Y, et al. A 1.22 TOPS and 1.52 mW/MHz augmented reality multicore processor with neural network NoC for HMD applications. IEEE J Solid State Circuits, 2015, 50(1), 113 doi: 10.1109/JSSC.2014.2352303
[6]
Lau J H. Recent advances and trends in advanced packaging. IEEE Trans Compon Packag Manuf Technol, 2022, 12(2), 228 doi: 10.1109/TCPMT.2022.3144461
[7]
Sharda J, Manley M, Kaul A, et al. Design and thermal analysis of 2.5D and 3-D integrated system of a CMOS image sensor and a sparsity-aware accelerator for autonomous driving. IEEE J Electron Devices Soc, 2024, 12, 426 doi: 10.1109/JEDS.2024.3354621
[8]
Chen Q M, Nie K M, Gao J, et al. A 1920 × 1080 array 2-D/3-D image sensor with 3-μs row-time single-slope ADC and 100-MHz demodulated PPD locked-In pixel. IEEE J Solid State Circuits, 2024, PP(99), 1
[9]
Wei S Y, Ning K, Kang L, et al. A real-time 2D/3D perception visual vector processor for 1920 × 1080 high-resolution high-speed intelligent vision chips. IEEE Trans Circuits Syst I Regul Pap, 2024, 71(2), 740 doi: 10.1109/TCSI.2023.3339784
[10]
Xu M M, Zhao M X, Yu S M, et al. Rounding shift channel post-training quantization using layer search. 2021 4th International Conference on Artificial Intelligence and Big Data (ICAIBD), 2021, 545
[11]
Zhao M X, Peng J B, Yu S M, et al. Exploring structural sparsity in CNN via selective penalty. IEEE Trans Circuits Syst Video Technol, 2022, 32(3), 1658 doi: 10.1109/TCSVT.2021.3071532
[12]
Zheng X M, Zhao M X, Luo Q, et al. A chip-level verification method for programmable vision chip based on deep learning algorithms. 2020 IEEE 5th International Conference on Integrated Circuits and Microsystems (ICICM), 2020, 281
[13]
Kim D, Lee S, Park D, et al. Indirect time-of-flight CMOS image sensor with on-chip background light cancelling and pseudo-four-tap/two-tap hybrid imaging for motion artifact suppression. IEEE J Solid State Circuits, 2020, 55(11), 2849 doi: 10.1109/JSSC.2020.3021246
[14]
Eki R, Yamada S, Ozawa H, et al. 9.6 A 1/2.3 inch 12.3 Mpixel with on-chip 4.97 TOPS/W CNN processor back-illuminated stacked CMOS image sensor. 2021 IEEE International Solid- State Circuits Conference (ISSCC), 2021, 154
[15]
Lepecq M, Dalgaty T, Fabre W, et al. End-to-end implementation of a convolutional neural network on a 3D-integrated image sensor with macropixel array. Sensors, 2023, 23(4), 1909 doi: 10.3390/s23041909
[16]
Sharda J, Li W T, Wu Q C, et al. Temporal frame filtering for autonomous driving using 3D-stacked global shutter CIS with IWO buffer memory and near-pixel compute. IEEE Trans Circuits Syst I Regul Pap, 2023, 70(5), 2074 doi: 10.1109/TCSI.2023.3252486

    • Search

    Advanced Search >>

    GET CITATION

    shu

    Export: BibTex EndNote

    Article Metrics

    Article views: 65 Times PDF downloads: 13 Times Cited by: 0 Times

    History

    Received: 22 January 2025 Revised: 20 March 2025 Online: Accepted Manuscript: 14 April 2025

    Catalog

      Email This Article

      User name:
      Email:*请输入正确邮箱
      Code:*验证码错误
      Send
      Article Navigation >  Journal of Semiconductors  > 2025  > Accepted Manuscript
      Siyuan Wei, Quanmin Chen, Jingyi Yu, Xuanzhe Xu, Yuxiao Wen, Runjiang Dou, Shuangming Yu, Guike Li, Kaiming Nie, Jie Cheng, Jiangtao Xu, Liyuan Liu, Nanjian Wu. A 2D/3D Vision Chip based on Organic Substrate 3D Package[J]. Journal of Semiconductors, 2025, In Press. doi: 10.1088/1674-4926/25010030 ****S Y Wei, Q M Chen, J Y Yu, X Z Xu, Y X Wen, R J Dou, S M Yu, G K Li, K M Nie, J Cheng, J T Xu, L Y Liu, and N J Wu, A 2D/3D Vision Chip based on Organic Substrate 3D Package[J]. J. Semicond., 2025, accepted doi: 10.1088/1674-4926/25010030
      Citation:
      Siyuan Wei, Quanmin Chen, Jingyi Yu, Xuanzhe Xu, Yuxiao Wen, Runjiang Dou, Shuangming Yu, Guike Li, Kaiming Nie, Jie Cheng, Jiangtao Xu, Liyuan Liu, Nanjian Wu. A 2D/3D Vision Chip based on Organic Substrate 3D Package[J]. Journal of Semiconductors, 2025, In Press. doi: 10.1088/1674-4926/25010030 ****
      S Y Wei, Q M Chen, J Y Yu, X Z Xu, Y X Wen, R J Dou, S M Yu, G K Li, K M Nie, J Cheng, J T Xu, L Y Liu, and N J Wu, A 2D/3D Vision Chip based on Organic Substrate 3D Package[J]. J. Semicond., 2025, accepted doi: 10.1088/1674-4926/25010030
      shu

      A 2D/3D Vision Chip based on Organic Substrate 3D Package

      DOI: 10.1088/1674-4926/25010030
      CSTR: 32376.14.1674-4926.25010030
      • 1.

        State Key Laboratory of Semiconductor Physics and Chip Technologies, Institute of Semiconductors, Chinese Academy of Sciences, Beijing 100083, China

      • 2.

        College of Materials Science and Optoelectronics Engineering, University of Chinese Academy of Sciences, Beijing 100049, China

      • 3.

        School of Microelectronics and Tianjin Key Laboratory of Imaging and Sensing Microelectronic Technology, Tianjin University, Tianjin 300072, China

      • 4.

        School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing 100049, China

      • 5.

        SuperPix Micro Technology Co., Ltd., Beijing 100085, China

      More Information
      • Siyuan Wei is currently pursuing a Ph.D. degree with the State Key Laboratory of Semiconductor Physics and Chip Technologies, Institute of Semiconductors, Chinese Academy of Sciences. His research interests include vision chips and hardware/software co-design
      • Quanmin Chen received the Ph.D. degree in microelectronics and solid electronics, Tianjin University, 2024. He is currently holding a post-doctoral position at Tianjin University. His current research interests focus on low-noise pixels and analog circuit design in time-of-flight range sensors
      • Jingyi Yu is currently pursuing the Ph.D. degree at the School of Microelectronics, Tianjin University, and at the State Key Laboratory of Semiconductor Physics and Chip Technologies, Institute of Semiconductors, Chinese Academy of Sciences. His research interests include CV algorithm design and implementation
      • Xuanzhe Xu is currently pursuing a Ph.D. degree with the State Key Laboratory of Semiconductor Physics and Chip Technologies, Institute of Semiconductors, Chinese Academy of Sciences. His research interests include vision chips, radar signal processing, and microprocessors
      • Yuxiao Wen is currently pursuing a Ph.D. degree with the State Key Laboratory of Semiconductor Physics and Chip Technologies, Institute of Semiconductors, Chinese Academy of Sciences
      • Runjiang Dou is a Senior Engineer at the Institute of Semiconductors, Chinese Academy of Sciences. His research interests include the design of vision systems based on intelligent hardware
      • Shuangming Yu is an Associate Professor with the State Key Laboratory of Semiconductor Physics and Chip Technologies, Institute of Semiconductors, CAS. His current research interests include vision chip design and ultra-low power circuit design
      • Guike Li has been an Assistant Professor of microelectronics and solid-state electronics with the State Key Laboratory of Semiconductor Physics and Chip Technologies, Institute of Semiconductors, Chinese Academy of Sciences. His current research interests include CMOS image sensors and integrated silicon photonics
      • Kaiming Nie has been a Professor with the School of Microelectronics, Tianjin University. His research interests include mixed analog/digital circuit design and CMOS image sensor design
      • Jie Cheng has been the deputy manager of SuperPix Micro Technology Co., Ltd. He is a professor-level senior engineer responsible for the digital circuit design of the company
      • Jiangtao Xu has been a Professor with the School of Microelectronics, Tianjin University. His research interests include CMOS image sensors and camera systems
      • Liyuan Liu joined the State Key Laboratory of Semiconductor Physics and Chip Technologies, Institute of Semiconductors, Chinese Academy of Sciences, Beijing, as an Associate Professor in 2012, where he became a professor in 2018. His research interests include mixed-signal IC design, CMOS image sensors design, terahertz image sensors design, and monolithic vision chip design
      • Nanjian Wu has been a Professor with the State Key Laboratory of Semiconductor Physics and Chip Technologies, Institute of Semiconductors, Chinese Academy of Sciences, Beijing, China, since 2000. His research includes the field of mixed-signal VLSI and vision chip design
      • Corresponding author: xujiangtao@tju.edu.cn; liuly@semi.ac.cn
      • Received Date: 2025-01-22
      • Revised Date: 2025-03-20
        • Available Online: 2025-04-14

      Catalog

        Journal of Semiconductors © 2017 All Rights Reserved   京ICP备05085259号-2

        /

        DownLoad:  Full-Size Img  PowerPoint
        Return
        Return