Citation: |
Jie Ma, Shenglong Zhuo, Lei Qiu, Yuzhu Gao, Yifan Wu, Ming Zhong, Rui Bai, Miao Sun, Patrick Yin Chiang. A review of ToF-based LiDAR[J]. Journal of Semiconductors, 2024, 45(10): 101201. doi: 10.1088/1674-4926/24040015
****
J Ma, S L Zhuo, L Qiu, Y Z Gao, Y F Wu, M Zhong, R Bai, M Sun, and P Y Chiang, A review of ToF-based LiDAR[J]. J. Semicond., 2024, 45(10), 101201 doi: 10.1088/1674-4926/24040015
|
A review of ToF-based LiDAR
DOI: 10.1088/1674-4926/24040015
CSTR: 32376.14.1674-4926.24040015
More Information-
Abstract
In recent years, propelled by the rapid iterative advancements in digital imaging technology and the semiconductor industry, encompassing microelectronic design, manufacturing, packaging, and testing, time-of-flight (ToF)-based imaging systems for acquiring depth information have garnered considerable attention from both academia and industry. This technology has emerged as a focal point of research within the realm of 3D imaging. Owing to its relatively straightforward principles and exceptional performance, ToF technology finds extensive applications across various domains including human−computer interaction, autonomous driving, industrial inspection, medical and healthcare, augmented reality, smart homes, and 3D reconstruction, among others. Notably, the increasing maturity of ToF-based LiDAR systems is evident in current developments. This paper comprehensively reviews the fundamental principles of ToF technology and LiDAR systems, alongside recent research advancements. It elucidates the innovative aspects and technical challenges encountered in both transmitter (TX) and receiver (RX), providing detailed discussions on corresponding solutions. Furthermore, the paper explores prospective avenues for future research, offering valuable insights for subsequent investigations.-
Keywords:
- time of flight,
- light detection and ranging,
- transmitter,
- receiver
-
References
[1] Sell J, O’Connor P. The xbox one system on a chip and kinect sensor. IEEE Micro, 2014, 34, 44 doi: 10.1109/mm.2014.9[2] Jiménez D, Pizarro D, Mazo M, et al. Modeling and correction of multipath interference in time of flight cameras. Image Vis Comput, 2014, 32, 1 doi: 10.1016/j.imavis.2013.10.008[3] Bamji C, Godbaz J, Oh M, et al. A review of indirect time-of-flight technologies. IEEE Trans Electron Devices, 2022, 69, 2779 doi: 10.1109/TED.2022.3145762[4] Chen J Y, Wang Y, Shen D Q, et al. Constant fraction discriminator for fast high-precision pulsed TOF laser rangefinder. International Symposium on Photoelectronic Detection and Imaging 2011: Sensor and Micromachined Optical Device Technologies, 2011, 8191, 382 doi: 10.1117/12.900690[5] Hutchings S W, Johnston N, Gyongy I, et al. A reconfigurable 3-D-stacked SPAD imager with In-pixel histogramming for flash LIDAR or high-speed time-of-flight imaging. IEEE J Solid-State Circuits, 2019, 54, 2947 doi: 10.1109/JSSC.2019.2939083[6] Camacho-Aguilera R E, Cai Y, Patel N, et al. An electrically pumped germanium laser. Opt Express, 2012, 20, 11316 doi: 10.1364/OE.20.011316[7] Bradley J D B, Su Z, Magden E S, et al. 1.8-μm thulium microlasers integrated on silicon. Optical Components and Materials XIII, 2016, 9744, 139 doi: 10.1117/12.2213678[8] Faist J, Capasso F, Sivco D L, et al. Quantum cascade laser. Science, 1994, 264, 553 doi: 10.1126/science.264.5158.553[9] Shtyrkova K, Callahan P T, Li N X, et al. Integrated CMOS-compatible Q-switched mode-locked lasers at 1900nm with an on-chip artificial saturable absorber. Opt Express, 2019, 27, 3542 doi: 10.1364/OE.27.003542[10] Dong P, Shafiiha R, Liao S R, et al. Wavelength-tunable silicon microring modulator. Opt Express, 2010, 18, 10941 doi: 10.1364/OE.18.010941[11] Rosenberg J C, Green W M J, Assefa S, et al. A 25 Gbps silicon microring modulator based on an interleaved junction. Opt Express, 2012, 20, 26411 doi: 10.1364/OE.20.026411[12] Alexander K, George J P, Verbist J, et al. Nanophotonic Pockels modulators on a silicon nitride platform. Nat Commun, 2018, 9, 3444 doi: 10.1038/s41467-018-05846-6[13] Zhu S, Zhong Q, Hu T, et al. Aluminum nitride ultralow loss waveguides and push-pull electro-optic modulators for near infrared and visible integrated photonics. 2019 Optical Fiber Communications Conference and Exhibition, IEEE, 2019, 1 doi: 10.1364/ofc.2019.w2a.11[14] Li S Y, Zhang D, Zhao J Y, et al. Silicon micro-ring tunable laser for coherent optical communication. Opt Express, 2016, 24, 6341 doi: 10.1364/OE.24.006341[15] Li N X, Timurdogan E, Poulton C V, et al. C-band swept wavelength erbium-doped fiber laser with a high-Q tunable interior-ridge silicon microring cavity. Optics express, 2016, 24, 22741 doi: 10.1364/OE.24.022741[16] Magden E S, Li N X, Raval M, et al. Transmissive silicon photonic dichroic filters with spectrally selective waveguides. Nat Commun, 2018, 9, 3009 doi: 10.1038/s41467-018-05287-1[17] Byrd M J, Timurdogan E, Su Z, et al. Mode-evolution-based coupler for high saturation power Ge-on-Si photodetectors. Opt Lett, 2017, 42, 851 doi: 10.1364/OL.42.000851[18] Ghosh S, Doerr C R, Piazza G. Aluminum nitride grating couplers. Appl Opt, 2012, 51, 3763 doi: 10.1364/AO.51.003763[19] Michel J, Liu J F, Kimerling L C. High-performance Ge-on-Si photodetectors. Nature Photon, 2010, 4, 527 doi: 10.1038/nphoton.2010.157[20] Going R, Seok T J, Loo J, et al. Germanium wrap-around photodetectors on Silicon photonics. Opt Express, 2015, 23, 11975 doi: 10.1364/OE.23.011975[21] Li N X, Xin M, Su Z, et al. A silicon photonic data link with a monolithic erbium-doped laser. Sci Rep, 2020, 10, 1114 doi: 10.1038/s41598-020-57928-5[22] Jankowski M, Langrock C, Desiatov B, et al. Ultrabroadband nonlinear optics in nanophotonic periodically poled lithium niobate waveguides. Optica, 2020, 7, 40 doi: 10.1364/OPTICA.7.000040[23] Singh N, Vermulen D, Ruocco A, et al. Supercontinuum generation in varying dispersion and birefringent silicon waveguide. Opt express, 2019, 27, 31698 doi: 10.1364/OE.27.031698[24] Leuthold J, Koos C, Freude W. Nonlinear silicon photonics. Nature Photon, 2010, 4, 535 doi: 10.1038/nphoton.2010.185[25] Frankis H C, Su Z, Li N, et al. Four-wave mixing in a high-Q aluminum oxide microcavity on silicon, CLEO: Science and Innovations. Optica Publishing Group, 2018, STh3I. 3 doi: 10.1364/CLEO_SI.2018.STh3I.3[26] Zhou B D, Xie D D, Chen S B, et al. Comparative analysis of SLAM algorithms for mechanical LiDAR and solid-state LiDAR. IEEE Sens J, 2023, 23, 5325 doi: 10.1109/JSEN.2023.3238077[27] Halterman R, Bruch M. Velodyne HDL-64E lidar for unmanned surface vehicle obstacle detection. SPIE Proceedings. Unmanned Systems Technology XII, 2010, 123 doi: 10.1117/12.850611[28] Kumagai O, Ohmachi J, Matsumura M, et al. 7.3 A 189 × 600 back-illuminated stacked SPAD direct time-of-flight depth sensor for automotive LiDAR systems. 2021 IEEE International Solid-State Circuits Conference (ISSCC), 2021, 110 doi: 10.1109/ISSCC42613.2021.9365961[29] Yang D H, Liu Y F, Chen Q J, et al. Development of the high angular resolution 360° LiDAR based on scanning MEMS mirror. Sci Rep, 2023, 13, 1540 doi: 10.1038/s41598-022-26394-6[30] Srowik A. 256 × 16 SPAD array and 16-channel ultrashort pulsed laser driver for automotive lidar. Int SPAD Workshop ISSW, 2020, 15, 18[31] Kabuk U. 4D solid-state lidar. In Int SPAD Workshop ISSW, 2020, 13, 17[32] Rogers C, Piggott A Y, Thomson D J, et al. A universal 3D imaging sensor on a silicon photonics platform. Nature, 2021, 590, 256 doi: 10.1038/s41586-021-03259-y[33] Long Z W, Fan X Y, He Z Y. Solid-state multi-beam scanning single-photon lidar system based on cascaded VIPA and SPAD array. 2022 Asia Communications and Photonics Conference (ACP), 2022, 2062 doi: 10.1109/ACP55869.2022.10088899[34] Wang Z H, Cao R, Sun Y L, et al. Ghost imaging of moving target based on the periodic pseudo-thermal light field generated by a 2D silicon OPA. IEEE Photonics J, 2022, 14, 6613408 doi: 10.1109/JPHOT.2022.3145000[35] Mabuchi Y, Manago N, Bagtasa G, et al. Multi-wavelength lidar system for the characterization of tropospheric aerosols and clouds. 2012 IEEE International Geoscience and Remote Sensing Symposium, IEEE, 2012, 2505 doi: 10.1109/IGARSS.2012.6351839[36] Abdollahi S, Marin-Palomo P, Ladouce M, et al. Programmable THz-range comb multiplication using a feedback-controlled multi-wavelength laser. 2023 Conference on Lasers and Electro-Optics Europe & European Quantum Electronics Conference (CLEO/Europe-EQEC), 2023, 1 doi: 10.1109/CLEO/Europe-EQEC57999.2023.10231859[37] Chen Y W, Li W, Hyyppä J, et al. A 10-nm spectral resolution hyperspectral LiDAR system based on an acousto-optic tunable filter. Sensors, 2019, 19, 1620 doi: 10.3390/s19071620[38] Shi X C, Lin J Y, Rao Y, et al. Gated-cross aggregation network for hyperspectral and LiDAR data classification. IEEE International Geoscience and Remote Sensing Symposium, 2023, 1265 doi: 10.1109/IGARSS52108.2023.10282184[39] Chiu C T, Ding Y C, Lin W C, et al. Chaos LiDAR based RGB-D face classification system with embedded CNN accelerator on FPGAs. IEEE Trans Circuits Syst I: Regul Pap, 2022, 69, 4847 doi: 10.1109/TCSI.2022.3190430[40] Wen S H, Wang T, Tao S. Hybrid CNN-LSTM architecture for LiDAR point clouds semantic segmentation. IEEE Robot Autom Lett, 2022, 7, 5811 doi: 10.1109/LRA.2022.3153899[41] Lee J S, Shin D, Jang B, et al. Single-chip beam scanner with integrated light source for real-time light detection and ranging. 2020 IEEE International Electron Devices Meeting (IEDM), 2020, 7.2.1 doi: 10.1109/IEDM13553.2020.9371987[42] Nakamura K, Narumi K, Kikuchi K, et al. Liquid-crystal tunable optical phased array for LiDAR applications. Smart Photonic and Optoelectronic Integrated Circuits XXIII, 2021, 94 doi: 10.1117/12.2591230[43] Juliano Martins R, Marinov E, Youssef M A B, et al. Metasurface-enhanced light detection and ranging technology. Nat Commun, 2022, 13, 5724 doi: 10.1038/s41467-022-33450-2[44] Marinov E, Martins R J, Ben Youssef M A, et al. Overcoming the limitations of 3D sensors with wide field of view metasurface-enhanced scanning lidar. Adv Photon, 2023, 5, 046005 doi: 10.1117/1.AP.5.4.046005[45] Kim I, Martins R J, Jang J, et al. Nanophotonics for light detection and ranging technology. Nat Nanotechnol, 2021, 16, 508 doi: 10.1038/s41565-021-00895-3[46] Kim S I, Park J, Jeong B G, et al. Electrically reconfigurable active metasurface for 3D distance ranging. 2020 IEEE International Electron Devices Meeting (IEDM), 2020, 7.1.1 doi: 10.1109/IEDM13553.2020.9372059[47] Zhuo S L, Xia T, Zhao L, et al. Solid-state dToF LiDAR system using an eight-channel addressable, 20-W/ch transmitter, and a 128 × 128 SPAD receiver with SNR-based pixel binning and resolution upscaling. IEEE J Solid State Circuits, 2023, 58, 757 doi: 10.1109/JSSC.2022.3227078[48] Piron F, Morrison D, Yuce M R, et al. A review of single-photon avalanche diode time-of-flight imaging sensor arrays. IEEE Sens J, 2021, 21, 12654 doi: 10.1109/JSEN.2020.3039362[49] Niclass C, Soga M, Matsubara H, et al. A 0.18-μm CMOS SoC for a 100-m-range 10-frame/s 200 × 96-pixel time-of-flight depth sensor. IEEE J Solid-State Circuits, 2014, 49, 315 doi: 10.1109/JSSC.2013.2284352[50] Villa F, Lussana R, Bronzi D, et al. CMOS imager with 1024 SPADs and TDCs for single-photon timing and 3-D time-of-flight. IEEE J Sel Top Quantum Electron, 2014, 20, 3804810 doi: 10.1109/JSTQE.2014.2342197[51] Bronzi D, Zou Y, Villa F, et al. Automotive three-dimensional vision through a single-photon counting SPAD camera. IEEE Trans Intell Transp Syst, 2016, 17, 782 doi: 10.1109/TITS.2015.2482601[52] Perenzoni M, Perenzoni D, Stoppa D. 64 × 64-pixel digital silicon photomultiplier direct ToF sensor with 100M photons/s/pixel background rejection and imaging/altimeter mode with 0.14% precision up to 6km for spacecraft navigation and landing. 2016 IEEE International Solid-State Circuits Conference (ISSCC), 2016, 118 doi: 10.1109/ISSCC.2016.7417935[53] Beer M, Schrey O M, Nitta C, et al. 1 × 80 pixel SPAD-based flash LIDAR sensor with background rejection based on photon coincidence. 2017 IEEE SENSORS, 2017, 1 doi: 10.1109/ICSENS.2017.8234048[54] Ximenes A R, Padmanabhan P, Lee M J, et al. A 256 × 256 45/65nm 3D-stacked SPAD-based direct TOF image sensor for LiDAR applications with optical polar modulation for up to 18.6dB interference suppression. 2018 IEEE International Solid-State Circuits Conference (ISSCC), 2018, 96 doi: 10.1109/ISSCC.2018.8310201[55] Zhang C, Lindner S, Antolovic I M, et al. A CMOS SPAD imager with collision detection and 128 dynamically reallocating TDCs for single-photon counting and 3D time-of-flight imaging. Sensors, 2018, 18, 4016 doi: 10.3390/s18114016[56] Ruokamo H, Hallman L, Rapakko H, et al. An 80 × 25 pixel CMOS single-photon range image sensor with a flexible on-chip time gating topology for solid state 3D scanning. ESSCIRC 2017-43rd IEEE European Solid State Circuits Conference. Leuven, Belgium. IEEE, 2017, 59 doi: 10.1109/ESSCIRC.2017.8094525[57] Zhang C, Lindner S, Antolović I M, et al. A 30-frames/s, 252 × 144 SPAD flash LiDAR with 1728 dual-clock 48.8-ps TDCs, and pixel-wise integrated histogramming. IEEE J Solid-State Circuits, 2019, 54, 1137 doi: 10.1109/JSSC.2018.2883720[58] Henderson R K, Johnston N, Hutchings S W, et al. 5.7 A 256 × 256 40nm/90nm CMOS 3D-stacked 120dB dynamic-range reconfigurable time-resolved SPAD imager. 2019 IEEE International Solid-State Circuits Conference (ISSCC), 2019, 106 doi: 10.1109/ISSCC.2019.8662355[59] Wang Y, Shi L L, Wang C J, et al. A direct TOF sensor with In-pixel differential time-to-charge converters for automotive flash LiDAR and other 3D applications. Proc Int Image Sens Workshop, 2019, R24[60] Kondo S, Kubota H, Katagiri H, et al. 5.1 A 240 × 192 pixel 10fps 70klux 225m-range automotive LiDAR SoC using a 40ch 0.0036mm2 voltage/time dual-data-converter-based AFE. 2020 IEEE International Solid-State Circuits Conference (ISSCC), 2020, 94 doi: 10.1109/ISSCC19947.2020.9063148[61] Morrison D, Kennedy S, Delic D, et al. A 64 × 64 SPAD flash LIDAR sensor using a triple integration timing technique with 1.95 mm depth resolution. IEEE Sens J, 2021, 21, 11361 doi: 10.1109/JSEN.2020.3030788[62] Jahromi S, Jansson J P, Keränen P, et al. A 32 × 128 SPAD-257 TDC receiver IC for pulsed TOF solid-state 3-D imaging. IEEE J Solid State Circuits, 2020, 55, 1960 doi: 10.1109/JSSC.2020.2970704[63] Zhang C, Zhang N, Ma Z J, et al. A 240 × 160 3D-stacked SPAD dToF image sensor with rolling shutter and In-pixel histogram for mobile devices. IEEE Open J Solid State Circuits Soc, 2021, 2, 3 doi: 10.1109/OJSSCS.2021.3118332[64] Padmanabhan P, Zhang C, Cazzaniga M, et al. 7.4 A 256 × 128 3D-stacked (45nm) SPAD FLASH LiDAR with 7-level coincidence detection and progressive gating for 100m range and 10klux background light. 2021 IEEE International Solid-State Circuits Conference (ISSCC), 2021, 111 doi: 10.1109/ISSCC42613.2021.9366010[65] Manuzzato E, Tontini A, Seljak A, et al. A 64 × 64-pixel flash LiDAR SPAD imager with distributed pixel-to-pixel correlation for background rejection, tunable automatic pixel sensitivity and first-last event detection strategies for space applications. 2022 IEEE International Solid-State Circuits Conference (ISSCC), 2022, 96 doi: 10.1109/ISSCC42614.2022.9731622[66] Kim M, Seo H, Kim S, et al. 6.11 A 320 × 240 CMOS LiDAR sensor with 6-transistor nMOS-only SPAD analog front-end and area-efficient priority histogram memory. 2024 IEEE International Solid-State Circuits Conference (ISSCC), 2024, 120 doi: 10.1109/ISSCC49657.2024.10454449[67] Zou C R, Ou Y Z, Zhu Y, et al. 6.8 A 256 × 192-pixel 30fps automotive direct time-of-flight LiDAR using 8 × current-integrating-based TIA, hybrid pulse position/width converter, and intensity/CNN-guided 3D inpainting. 2024 IEEE International Solid-State Circuits Conference (ISSCC), 2024, 114 doi: 10.1109/ISSCC49657.2024.10454461[68] Hu J, Liu B Z, Ma R, et al. A 32 × 32-pixel flash LiDAR sensor with noise filtering for high-background noise applications. IEEE Trans Circuits Syst I: Regul Pap, 2022, 69, 645 doi: 10.1109/TCSI.2020.3048367[69] Jo S, Kong H J, Bang H, et al. High resolution three-dimensional flash LIDAR system using a polarization modulating Pockels cell and a micro-polarizer CCD camera. Opt Express, 2016, 24, A1580 doi: 10.1364/OE.24.0A1580[70] Wang D K, Watkins C, Aradhya M, et al. A large aperture 2-axis electrothermal MEMS mirror for compact 3D LiDAR. 2019 International Conference on Optical MEMS and Nanophotonics (OMN), 2019, 180 doi: 10.1109/OMN.2019.8925039[71] Shen Z C, Zhao F, Jin C Q, et al. Monocular metasurface camera for passive single-shot 4D imaging. Nat Commun, 2023, 14, 1035 doi: 10.1038/s41467-023-36812-6[72] Han S H, Park S, Chun J H, et al. 6.7 A 160 × 120 flash LiDAR sensor with fully analog-assisted in- pixel histogramming TDC based on self-referenced SAR ADC. 2024 IEEE International Solid-State Circuits Conference (ISSCC), 2024, 112 doi: 10.1109/ISSCC49657.2024.10454470[73] Chen G B, Wiede C, Kokozinski R. Data processing approaches on SPAD-based d-TOF LiDAR systems: A review. IEEE Sens J, 2021, 21, 5656 doi: 10.1109/JSEN.2020.3038487 -
Proportional views