1. Introduction

The real world owns distance information in a three dimensional coordinate system, and most creatures on earth have evolved binocular or compound eyes to gain depth information^[1]. However, a conventional image sensor purely captures flat images and neglects the distance information in the depth dimension, which plays an important role in many applications, such as aerospace exploration, automobile assistance, industrial detection and high-speed tracking.

In the past few decades, 3D imaging has been regarded as one of the most interesting and innovative research fields. A number of different depth image acquisition methods have been presented. Among these methods, the passive method refers to acquiring depth information by using multi-view image data, according to the triangulation measurement principle^[2]. The advantages of this method include high resolution and the lack of a need for illumination, while the disadvantage is the computational expense. On the other hand, the active method refers to optical distance measurement techniques, including interferometry, triangulation and time-of-flight (ToF)^[3]. In recent years a new generation of depth measurement methods, called ToF cameras, has been developed. ToF cameras capture a depth image by measuring the transmission time of the emitted light along the reflection path between each pixel and its corresponding point in a scene^{[4, 5]}. Compared to traditional 3D image acquisition methods, ToF cameras are able to provide distance measurement results for each pixel at the video frame rate with compact setup size and reduced power consumption^[6].

Depth measurement methods fulfilling the principles of ToF cameras can be further classified into three categories, namely: the pulsed modulation method^{[7, 8]}, the continuous-wave (CW) modulation method^{[9, 10]}, and the digital phase demodulation method based on the single-photon synchronous detection (SPSD)^[11-13]. Approaches based on the CW modulation method have the advantage of high linearity while the pulsed modulation method enables fast depth measurement processing and an increased signal-to-noise ratio^[3]. The SPSD utilizes single-photon avalanche diodes (SPADs) as signal-photon detectors and shows better measurement performance in accuracy and the measuring range^{[14, 15]}. However, the disadvantages of SPSD are that it requires a complex circuit and imaging system design, it also has a limited pixel resolution.

This paper presents a 256 × 256 image sensor controller that is compatible for both of the 2D image exposure and the 3D depth measurement functions. It generates control signals for 3D depth measurement based on the continuous-wave modulation method. An refined circuit design technique in the logic level is utilized to generate rolling shutter control signals in a more efficient way. Critical parameters, such as the exposure time, the modulation frequency, and the frame rate, are able to be easily tuned inside the controller.

2. Principle of operation

We designed the image sensor controller based on the earliest presented and the most widely adopted ToF measurement principle, namely the CW modulation method^{[4, 5]}. The principle of the CW modulation method is illustrated in Fig. 1. The measurement setup is composed of two separate parts: one is the modulated light emitter, and the other is the light receiver. The light emitter is typically made by a near-infrared laser or a light emitting diode (LED) array with a peak wavelength near 850 nm, which is used to achieve better angular resolution performance compared to micro-and ultrasonic-waves^[3]. The light emitter is driven by a sinusoidal wave signal with a frequency of around 20 MHz, corresponding to an ambiguity-free measurement range of about 7.5 m. On the other hand, the light receiver is made up of a CCD or CMOS image sensor, or digital single-photon detectors. The earliest image sensors based on the CW modulation method were fabricated with the CCD process^{[4, 5]}. Later on, References ^{[9, 11]} have proven that this method could also be realized with the CMOS image sensor and the SPAD.

In the CW modulation method, each pixel in the receiver samples the reflected light in a massively parallel fashion. In every period cycle, each pixel samples four times at equal intervals, with 0°, 90°, 180°, and 270° phase shifts, getting four sampling values S₀, S₁, S₂, and S₃, as shown in Fig. 1. Then, the phase delay $\phi_{\rm d}$ between the emitted and reflected light is calculated by

$\label{eq1} \phi _{\rm d} =\arctan \frac{S_3 -S_1 }{S_2 -S_0}.$

(1)

The offset of the sampling values is calculated by

$\label{eq2} I_{\rm offset} =\frac{S_0 +S_1 +S_2 +S_3 }{4}.$

(2)

The amplitude of the sampling values is calculated by

$\label{eq3} I_{\rm amp} =\frac{\sqrt {[S_3-S_1]^2+[S_2-S_0]^2} }{2}.$

(3)

The amplitude to offset ratio helps predict the quality of the measurement result.

The time delay t_d is able to be calculated in further according to the phase delay

$\label{eq4} t_{\rm d}=\frac{\phi _{\rm d}}{2\pi f_{\rm m}},$

(4)

where $f_{\rm m}$ is the modulation frequency of the emitted light.

Since the emitted light transmits along a straight line in free space, the distance between a particular pixel and its corresponding point in the scene equals half of the flight distance

$\label{eq5} D=\frac{ct_{\rm d}}{2}=\frac{c\phi _{\rm d}}{4\pi f_{\rm m}},$

(5)

where $c$ refers to the speed of light in vacuum.

It should be mentioned that the ambiguity-free measurement range is inversely proportional to the modulation frequency

$\label{eq6} D_{\rm range} f_{\rm m} =\frac{c}{2}\approx 1.5\times 10^8\, {\rm (m/s)}=150 \, ({\rm m}\cdot {\rm MHz}).$

(6)

The modulation frequency is limited in two aspects. On the one hand, the maximum modulation frequency is limited to around 25 MHz because of the LED dynamic characteristics; that is, a rising and falling time of 12 ns. On the other hand, the minimum modulation frequency is limited to around 15 MHz, which corresponds to the maximum measurement range of 10 m, as determined by the intensity of the reflected light and the sensitivity of the image sensor. According to these limitations, we set the modulation frequency to 20 MHz in our test, which corresponds to an ambiguity-free measurement range of 7.5 m.

3. Circuit design and implementation

3.1 Pixel structure

Figure 2 shows the schematic of the four transistor active pixel sensor (4T-APS)^[16]. We chose to use a pinned photodiode (PPD) as the structure of the image sensor pixel for better noise performance. Figure 3 shows the cross-section structure of the proposed pixel. The junction depth is carefully designed by optimizing the tradeoff between the transfer speed and the absorption coefficient. The physical structure of the pixel is split into two symmetrical parts to enhance the lateral drift field, which results in a higher transfer speed. According to the simulation result, when the transfer transistor TX is on and the transfer transistor TXD is off, the photoelectrons are transferred from the PPD to the float diffusion (FD) within 10 ns. When the TX is off and the TXD is on, the photoelectrons are transferred from the PPD to the drain (DR) to be cleared.

Figure 4 shows the timing diagram for measuring a frame in the 3D mode. The LED is driven by the 20 MHz modulation signal. The control timing of the 3D frame capture process contains four separate parts, with the TX sampling phase shifts of 0°, 90°, 180°, and 270°, respectively. Each part consists of two steps: the integration process and the reading out process. In the integration process, control signals TX and TXD are set in a complementary manner with a modulation frequency of 20 MHz. The number of TX or TXD pulses is set by a reconfigurable parameter that is stored in the image sensor controller. In the reading out process, the integrated voltage is sampled by the auxiliary circuit and the sampled data is stored for further calculation according to the previous formulae.

It should be mentioned that Figure 4 only shows the exposure process of one specific row and the RST signal is not shown for simplicity. The image sensor controller generates control signals for each row in a rolling shutter manner, and the RST signals are correspondingly generated to realize the correlated double sampling (CDS) in the digital domain.

In the conventional image sensing mode, the control timing contains two steps. In the first step, the TX is off and the TXD and the RST is on. In this case, the PPD is reset and the reset voltage at the FD is read out for the CDS operation. In the second step, the TX turns on and the TXD turns off, so that the integral signal at the PPD is read out. The control signals are generated in a rolling shutter manner, and the widths and intervals of the control signals are defined as configurable parameters.

3.3 Sensor architecture

Figure 5 shows the architecture of the image sensor, which contains a 256 × 256 pixel array. The image sensor controller generates the control signals for each row of pixels, and the control signals are able to be reconfigured for either the conventional image sensing mode or the depth measurement mode to realize the rolling shutter exposure and the CDS operation.

The image sensor controller generates 256 groups of control signals in parallel, while each group controls one specific row of pixels. In the depth measurement mode, the image sensor controller captures images frame after frame, with periodically varying phase shifts. In each frame, it samples 256 rows of signals in a rolling shutter manner, with the same time delay between neighboring rows. Figure 5 shows two neighboring groups of control signals in the depth measurement mode. In order to make the generated control signals flexibly tuned, we have defined a set of configurable parameters, such as the pulse width of the RST signal, the pulse width of the TX and TXD control signals, the row exposure delay from one rising edge to the next rising edge of the RST signal, the time delay from the RST to the TX signal, and the number of TX pulses for each row of pixels.

3.4 Reconfigurable controller}

Figure 6(a) shows the simplified schematic diagram of the rolling shutter control signals in the depth measurement mode. The control signals contain 256 paths, corresponding to 256 rows of pixels. The control signals are synchronized with one another, and the delay time between neighboring control signals remains the same as a reconfigurable value. In order to make these control signals flexibly tuned, we introduce three reconfigurable parameters, namely: the signal length t_p, the time period t₀ of each pulse cycle, and the time delay t_d between neighboring control signals.

Usual implementation of the image sensor controller requires at least two counters for each signal path to generate the control signals. One counter is utilized to count the clock periods number of the signal level, while the other counter adjusts the control signal length t_p and the time delay t_d. In such an implementation, 256 paths of control signals are treated separately and each path utilizes one independent 16-bit counter to count the pulse width. Although this implementation is simple and intuitive, it costs at least 256 16-bit counters, which leads to large chip area and power consumption.

Figure 6(b) illustrates the refined circuit design technique to reduce the hardware overhead, which solves the previously presented problem. In this technique, we utilized a 16-bit counter to generate the periodic pulse signal and we designed two additional counters to generate the strobe signal for each row in the rolling shutter manner. In addition, we introduced two-input logic AND gates to combine the periodic pulse signal and the strobe signals. The expected rolling shutter control signals are generated as shown in Fig. 6(a). Compared to the usual implementation, 256 16-bit counters designed for separate control signals are saved.

In order to prove the benefits of the proposed method, we utilized the Design Compiler tool to synthesize the usual implementation of the image sensor controller for comparison. As the synthesis result shows, the chip area of the usual implementation is 2.71 mm² and the power consumption estimated by the synthesis tool is 129 mW. On the other hand, the image sensor controller adopting the refined technique occupies only 1.10 mm² and the power consumption is only 64.9 mW. The synthesized result shows that the refined circuit costs 59.2% less chip area and 49.6% less power consumption compared to the usual implementation, while the circuit function is properly remained.

3.5 Design implementation

The image sensor controller is designed with two reconfigurable measurement modes, namely the 2D and 3D measurement modes. Either of the measurement modes is able to be switched to the other mode according to the external control signal Sensor_mode. Reconfigurable parameters are stored in a separate register file, corresponding to either of the measurement modes. Figure 7 shows the parameter register files with data and control paths connecting with them.

In order to make the control signals flexible tuned, we have defined two groups of reconfigurable parameters for the image sensor controller, with 18 parameters for each group. Table 1 lists all of the parameters related to the RST signal in either of the measurement modes. The bit widths, default values and brief descriptions of these parameters are also shown.

Figure 8 shows the state machine of the image sensor controller. When the controller is started or reset, it first enters the IDLE mode, in which all of the reconfigurable parameters are reset to default values. When the START signal comes, the controller decides its next state according to the external control signal Sensor_mode. If the Sensor_mode is at a high level, the controller enters the 3D mode and starts to generate control signals for the 3D depth measurement. On the other hand, if the Sensor_mode is at a low level, the controller turns into the 2D mode and starts to generate control signals for the conventional 2D image sensing.

4. Test results

The image sensor controller presented in this paper is designed in a GSMC 0.18 μm 1P5M standard CMOS technology. Figure 9 shows the microphotograph of the image sensor controller. The size of the image sensor controller is 700 × 3380 μm². The power consumption estimated by the synthesis tool is 65 mW under a 1.8 V supply voltage and 100 MHz clock frequency.

We utilized the VCS tool for the post-layout simulation. Our simulation results show that the proposed image sensor controller functions properly in either of the measurement modes under a 100 MHz clock frequency.

The test platform designed for the fabricated chip is shown in Fig. 10. The chip under test outputs raw integral signals in the analog domain. The output signals are sampled by 16 channels of 14-bit A/D converters, followed by parallel to serial conversion components. A Cyclone Ⅲ FPGA is utilized to communicate with the chip under test and to process the sampled data. The test platform also provides the 80 MHz clock, 1.8 V and 3.3 V supply voltages, and 1.0 V and 2.0 V bias voltages for the chip under test, and the test platform is powered merely through a universal USB interface.

We tested the image sensor controller in either of the measurement modes. Figure 11 shows the waveform of the output signals in the 3D measurement mode. The horizontal synchronization signal Hsync is generated by the sensor controller, which helps to indicate that the output signal of the current row is ready to be sampled. In each row exposure cycle, the sudden decrease of the raw signal after the first Hsync signal indicates that the charged voltage at FD is read out. The raw signal turns back to a high level soon after the second Hsync signal arrives, which happens because the reset voltage at FD is read out for the realization of the CDS in the digital domain. The slight drop of the raw signal after reset is caused by the turn-off operation of an internal strobe signal.

The parameter configuration function of the proposed image sensor controller is also verified under the test platform. We captured the output configured parameters from the chip under test, as shown in Fig. 12(a). Meanwhile, the waveform of the parameter sequence is generated with the Matlab in Fig. 12(b). It is seen that the test result shown in Fig. 12(a) and the simulation results shown in Fig. 12(b) are the same.

5. Conclusion

An image sensor controller that is compatible for depth measurement has been proposed in this paper. The image sensor controller is able to generate control signals not only for the conventional image sensing mode (2D) but also for the depth measurement mode (3D), based on the phase-modulated time-of-flight method. In addition, this paper presents a refined circuit design technique for generating rolling shutter control signals in the depth measurement mode. Compared to the usual implementation, the presented image sensor controller saves 59.4% circuit area and 49.6% power consumption. The image sensor is fabricated using standard 0.18 $\mu$m CMOS technology. Our test results show that the controller functions properly.