Time-of-flight camera

From Wikipedia, the free encyclopedia

A time-of-flight camera (ToF camera) is a range imaging camera system that resolves distance based on the known speed of light, measuring the time-of-flight of a light signal between the camera and the subject for each point of the image. The time-of-flight camera is a class of scannerless LIDAR, in which the entire scene is captured with each laser or light pulse, as opposed to point-by-point with a laser beam such as in scanning LIDAR systems.[1]

Time-of-flight camera products for civil applications began to emerge around 2000,[2] as the semiconductor processes became fast enough for such devices. The systems cover ranges of a few centimeters up to about 60 m. The distance resolution is about 1 cm. The lateral resolution of time-of-flight cameras is generally low compared to standard 2D video cameras, with most commercially available devices at 320 × 240 pixels or less as of 2011.[3][4][5][6] Compared to 3D laser scanning methods for capturing 3D images, TOF cameras operate very quickly, providing up to 160 images per second.[7]

Types of devices

Several different technologies for time-of-flight cameras have been developed.

RF-modulated light sources with phase detectors

Photonic Mixer Devices (PMD),[8] the Swiss Ranger, and CanestaVision[9] work by modulating the outgoing beam with an RF carrier, then measuring the phase shift of that carrier on the receiver side. This approach has a modular error challenge; ranges are mod the maximum range, which is the RF carrier wavelength. The Swiss Ranger is a compact, short-range device, with ranges of 5 or 10 meters, with 176 x 144 pixels. With phase unwrapping algorithms, the maximum uniqueness range can be increased. The PMD can provide ranges up to 60m. Illumination is pulsed LEDs, rather than a laser.[10] CanestaVision developer Canesta was purchased by Microsoft in 2010.

Range gated imagers

These devices have a built-in shutter in front of the image sensor that opens and closes at the same rate as the light pulses are sent out. Because part of every returning pulse is blocked by the shutter according to its time of arrival, the amount of light received relates to the distance the pulse has traveled. The distance can be calculated using the equation, z = R (S2 − S1) / 2(S1 + S2) + R / 2 for an ideal camera. R is the camera range, determined by the round trip of the light pulse, S1 the amount of the light pulse that is received, and S2 the amount of the light pulse that is blocked.[11][12]

The ZCam by 3DV Systems[1] is a range-gated system. Microsoft purchased 3DV in 2009. Microsoft's second-generation Kinect sensor was developed using knowledge gained from Canesta and 3DV Systems.[13]

Similar principles are used in the ToF camera line developed by the Fraunhofer Institute of Microelectronic Circuits and Systems and TriDiCam. These cameras employ photodetectors with a fast electronic shutter.

The depth resolution of ToF cameras can be improved with ultra-fast gating intensified CCD cameras. These cameras provide gating times down to 200ps and enable ToF setup with sub-millimeter depth resolution. [14]

Range gated imagers can also be used in 2D imaging to suppress anything outside a specified distance range, such as to see through fog. A pulsed laser provides illumination, and an optical gate allows light to reach the imager only during the desired time period.[15][16]

Components

A time-of-flight camera consists of the following components:

  • Illumination unit: It illuminates the scene. As the light has to be modulated with high speeds up to 100 MHz, only LEDs or laser diodes are feasible. The illumination normally uses infrared light to make the illumination unobtrusive.
  • Optics: A lens gathers the reflected light and images the environment onto the image sensor. An optical band-pass filter only passes the light with the same wavelength as the illumination unit. This helps suppress background light.
  • Image sensor: This is the heart of the TOF camera. Each pixel measures the time the light has taken to travel from the illumination unit to the object and back. Several different approaches are used for timing; see types of devices above.
  • Driver electronics: Both the illumination unit and the image sensor have to be controlled by high speed signals. These signals have to be very accurate to obtain a high resolution. For example, if the signals between the illumination unit and the sensor shift by only 10 picoseconds, the distance changes by 1.5 mm. For comparison: current CPUs reach frequencies of up to 3 GHz, corresponding to clock cycles of about 300 ps - the corresponding 'resolution' is only 45 mm.
  • Computation/Interface: The distance is calculated directly in the camera. To obtain good performance, some calibration data is also used. The camera then provides a distance image over a USB or Ethernet interface.

Principle

Diagrams illustrating the principle of a time-of-flight camera with analog timing

The simplest version of a time-of-flight camera uses light pulses. The illumination is switched on for a very short time, the resulting light pulse illuminates the scene and is reflected by the objects. The camera lens gathers the reflected light and images it onto the sensor plane. Depending on the distance, the incoming light experiences a delay. As light has a speed of approximately c = 300,000,000 meters per second, this delay is very short: an object 2.5 m away will delay the light by:

t_{D}=2\cdot {\frac Dc}=2\cdot {\frac {2.5\;{\mathrm {m}}}{300\;000\;000\;{\frac {{\mathrm {m}}}{{\mathrm {s}}}}}}=0.000\;000\;016\;66\;{\mathrm {s}}=16.66\;{\mathrm {ns}}[17]

The pulse width of the illumination determines the maximum range the camera can handle. With a pulse width of e.g. 50 ns, the range is limited to

D_{{\mathrm {max}}}={\frac {1}{2}}\cdot c\cdot t_{0}={\frac {1}{2}}\cdot 300\;000\;000\;{\frac {{\mathrm {m}}}{{\mathrm {s}}}}\cdot 0.000\;000\;05\;{\mathrm {s}}=\!\ 7.5\;{\mathrm {m}}

These short times show that the illumination unit is a critical part of the system. Only with some special LEDs or lasers is it possible to generate such short pulses.

The single pixel consists of a photo sensitive element (e.g. a photo diode). It converts the incoming light into a current. In analog timing imagers, connected to the photo diode are fast switches, which direct the current to one of two (or several) memory elements (e.g. a capacitor) that act as summation elements. In digital timing imagers, a time counter, running at several gigahertz, is connected to each photodetector pixel and stops counting when light is sensed.

In the diagram of an analog timer, the pixel uses two switches (G1 and G2) and two memory elements (S1 and S2). The switches are controlled by a pulse with the same length as the light pulse, where the control signal of switch G2 is delayed by exactly the pulse width. Depending on the delay, only part of the light pulse is sampled through G1 in S1, the other part is stored in S2. Depending on the distance, the ratio between S1 and S2 changes as depicted in the drawing.[9] Because only small amounts of light hit the sensor within 50 ns, not only one but several thousand pulses are sent out (repetition rate tR) and gathered, thus increasing the signal to noise ratio.

After the exposure, the pixel is read out and the following stages measure the signals S1 and S2. As the length of the light pulse is defined, the distance can be calculated with the formula:

D={\frac {1}{2}}\cdot c\cdot t_{0}\cdot {\frac {S2}{S1+S2}}

In the example, the signals have the following values: S1 = 0.66 and S2 = 0.33. The distance is therefore:

D=7.5\;{\mathrm {m}}\cdot {\frac {0.33}{0.33+0.66}}=2.5\;{\mathrm {m}}

In the presence of background light, the memory elements receive an additional part of the signal. This would disturb the distance measurement. To eliminate the background part of the signal, the whole measurement can be performed a second time with the illumination switched off. If the objects are further away than the distance range, the result is also wrong. Here, a second measurement with the control signals delayed by an additional pulse width helps to suppress such objects. Other systems work with a sinusoidally modulated light source instead of the pulse source.

Advantages

Simplicity

In contrast to stereo vision or triangulation systems, the whole system is very compact: the illumination is placed just next to the lens, whereas the other systems need a certain minimum base line. In contrast to laser scanning systems, no mechanical moving parts are needed.

Efficient distance algorithm

It is very easy to extract the distance information out of the output signals of the TOF sensor, therefore this task uses only a small amount of processing power, again in contrast to stereo vision, where complex correlation algorithms have to be implemented. After the distance data has been extracted, object detection, for example, is also easy to carry out because the algorithms are not disturbed by patterns on the object.

Speed

Time-of-flight cameras are able to measure the distances within a complete scene with one shot. As the cameras reach up to 160 frames per second, they are ideally suited to be used in real-time applications.

Disadvantages

Background light

Although most of the background light coming from artificial lighting or the sun is suppressed, the pixel still has to provide a high dynamic range. The background light also generates electrons, which have to be stored. For example, the illumination units in today's TOF cameras can provide an illumination level of about 1 watt. The Sun has an illumination power of about 50 watts per square meter after the optical band-pass filter. Therefore, if the illuminated scene has a size of 1 square meter, the light from the sun is 50 times stronger than the modulated signal.

Interference

If several time-of-flight cameras are running at the same time, the cameras may disturb each other's measurements. There exist several possibilities for dealing with this problem:

  • Time multiplexing: A control system starts the measurement of the individual cameras consecutively, so that only one illumination unit is active at a time.
  • Different modulation frequencies: If the cameras modulate their light with different modulation frequencies, their light is collected in the other systems only as background illumination but does not disturb the distance measurement.

Multiple reflections

In contrast to laser scanning systems, where only a single point is illuminated at once, the time-of-flight cameras illuminate a whole scene. Due to multiple reflections, the light may reach the objects along several paths and therefore, the measured distance may be greater than the true distance.

Applications

Range image of a human face captured with a time-of-flight camera

Automotive applications

Time-of-flight cameras are used in assistance and safety functions for advanced automotive applications such as active pedestrian safety, precrash detection and indoor applications like out-of-position (OOP) detection.[18][19]

Human-machine interfaces and gaming

As time-of-flight cameras provide distance images in real time, it is easy to track movements of humans. This allows new interactions with consumer devices such as televisions. Another topic is to use this type of cameras to interact with games on video game consoles.[20] The second-generation Kinect sensor which is a standard component of the Xbox One console uses a time-of-flight camera for its range imaging,[21] enabling natural user interfaces and gaming applications using computer vision and gesture recognition techniques. Creative and Intel also provide a similar type of interactive gesture time-of-flight camera for gaming, the Senz3D based on the DepthSense 325 camera of Softkinetic.[22] Infineon and pmdtechnologies enable tiny integrated 3D depth cameras for close-range gesture control of consumer devices like all-in-one PCs and laptops.[23]

Measurement and machine vision

Range image with height measurements

Other applications are measurement tasks, e.g. for the fill height in silos. In industrial machine vision, the time-of-flight camera helps to classify objects and help robots find the items, for instance on a conveyor. Door controls can distinguish easily between animals and humans reaching the door.

Robotics

Another use of these cameras is the field of robotics: Mobile robots can build up a map of their surroundings very quickly, enabling them to avoid obstacles or follow a leading person. As the distance calculation is simple, only little computational power is used.

Brands

Active brands (as of 2011)
  • D-IMager - TOF camera by Panasonic Electric Works[24]
  • DepthSense - TOF cameras and modules, including RGB sensor and microphones by SoftKinetic[25]
  • Fotonic - TOF cameras and software powered by Panasonic CMOS chip[26]
  • IRMA MATRIX - TOF camera, used for automatic passenger counting on mobile and stationary applications by iris-GmbH[27]
  • Kinect - hands-free user interface platform by Microsoft for video game consoles and PCs, using time-of-flight cameras in its second generation of sensor devices.[21]
  • pmd - camera reference designs and software (pmd[vision], including TOF modules [CamBoard]) and TOF imagers (PhotonICs) by pmdtechnologies[28]
  • real.IZ 2+3D - High-resolution SXGA (1280×1024) TOF camera developed by startup company odos imaging, integrating conventional image capture with TOF ranging in the same sensor. Based on technology developed at Siemens.[29]
  • Senz3D - TOF camera by Creative and Intel based on DepthSense 325 camera of Softkinetic, used for gaming.[22]
  • SwissRanger - an industrial TOF-only camera line originally by the Centre Suisse d'Electronique et Microtechnique, S.A. (CSEM), now developed by the spin out company Mesa Imaging[30]
  • 3D MLI Sensor - TOF imager, modules, cameras, and software by IEE (International Electronics & Engineering), based on modulated light intensity (MLI)[31]
  • TOFCam Stanley - TOF camera by Stanley Electric[32]
  • TriDiCam - TOF modules and software, the TOF imager originally developed by Fraunhofer Institute of Microelectronic Circuits and Systems, now developed by the spin out company TriDiCam[33]
Defunct brands
  • CanestaVision[34] - TOF modules and software by Canesta (company acquired by Microsoft in 2010)
  • OptriCam - TOF cameras and modules by Optrima (rebranded DepthSense prior to SoftKinetic merger in 2011)
  • ZCam - TOF camera products by 3DV Systems, integrating full-color video with depth information (assets sold to Microsoft in 2009)

  • D-IMager by Panasonic

  • pmd[vision] CamCube by pmdtechnologies

  • SwissRanger 4000 by MESA Imaging

  • FOTONIC-B70 by Fotonic

  • Argos3D-P100 by Bluetechnix

  • 3D MLI Sensor by IEE S.A.

  • ARTTS camera prototype

  • pmd[vision] CamBoard by pmdtechnologies

  • Kinect for Xbox One by Microsoft

See also

References

  1. ↑ 1.0 1.1 Iddan, Gavriel J.; Yahav, Giora (2001-01-24). "3D imaging in the studio (and elsewhere…)". Proceedings of SPIE 4298 (San Jose, CA: SPIE, published 2003-04-29). p. 48. doi:10.1117/12.424913. Archived from the original on 2009-06-12. Retrieved 2009-08-17. "The [time-of-flight] camera belongs to a broader group of sensors known as scanner-less LIDAR (i.e. laser radar having no mechanical scanner); an early [1990] example is [Marion W.] Scott and his followers at Sandia." 
  2. ↑ "Product Evolution". 3DV Systems. Archived from the original on 2009-02-28. Retrieved 2009-02-19. "Z-Cam, the first depth video camera, was released in 2000 and was targeted primarily at broadcasting organizations." 
  3. ↑ Schuon, Sebastian; Theobalt, Christian; Davis, James; Thrun, Sebastian (2008-07-15). "High-quality scanning using time-of-flight depth superresolution". IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2008. Anchorage, Alaska: Institute of Electrical and Electronics Engineers. pp. 1–7. doi:10.1109/CVPRW.2008.4563171. ISBN 978-1-4244-2339-2. Retrieved 2009-07-31. "The Z-cam can measure full frame depth at video rate and at a resolution of 320×240 pixels." 
  4. ↑ "Canesta's latest 3D Sensor - "Cobra" ... highest res CMOS 3D depth sensor in the world" (Flash Video). Sunnyvale, California: Canesta. 2010-10-25. "Canesta "Cobra" 320 x 200 Depth Sensor, capable of 1mm depth resolution, USB powered, 30 to 100 fps […] The complete camera module is about the size of a silver dollar" 
  5. ↑ SR4000 Data Sheet (Rev 2.6 ed.). Zürich, Switzerland: Mesa Imaging. August 2009. p. 1. Retrieved 2009-08-18. "176 x 144 pixel array (QCIF)" 
  6. ↑ PMD[vision] CamCube 2.0 Datasheet (No. 20090601 ed.). Siegen, Germany: PMDTechnologies. 2009-06-01. p. 5. Archived from the original on 2012-02-25. Retrieved 2009-07-31. "Type of Sensor: PhotonICs PMD 41k-S (204 x 204)" 
  7. ↑ http://ww2.bluetechnix.com/en/products/argos3d/
  8. ↑ Christoph Heckenkamp: Das magische Auge - Grundlagen der Bildverarbeitung: Das PMD Prinzip. In: Inspect. Nr. 1, 2008, S. 25–28.
  9. ↑ 9.0 9.1 Gokturk, Salih Burak; Yalcin, Hakan; Bamji, Cyrus (24 January 2005). "A Time-Of-Flight Depth Sensor - System Description, Issues and Solutions" (pdf). IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2004 (Institute of Electrical and Electronics Engineers): 35–45. doi:10.1109/CVPR.2004.291. Archived from the original on 2007-06-23. Retrieved 2009-07-31. "The differential structure accumulates photo-generated charges in two collection nodes using two modulated gates. The gate modulation signals are synchronized with the light source, and hence depending on the phase of incoming light, one node collects more charges than the other. At the end of integration, the voltage difference between the two nodes is read out as a measure of the phase of the reflected light." 
  10. ↑ "Mesa Imaging - Products". August 17, 2009. 
  11. ↑ Medina A, Gayá F, and Pozo F. Compact laser radar and three-dimensional camera. 23 (2006). J. Opt. Soc. Am. A. pp. 800–805. 
  12. ↑ Medina, Antonio. Three Dimensional Camera and Rangefinder. January 1992. United States Patent 5081530. 
  13. ↑ http://www.pcworld.com/article/2042958/kinect-for-windows-developers-kit-slated-for-november-adds-green-screen-technology.html
  14. ↑ "Submillimeter 3-D Laser Radar for Space Shuttle Tile Inspection.pdf". 
  15. ↑ http://www.laseroptronix.se/gated/sealynx.pdf
  16. ↑ "Sea-Lynx Gated Camera - active laser camera system". 
  17. ↑ "CCD/CMOS Lock-In Pixel for Range Imaging: Challenges, Limitations and State-of-the-Art" - CSEM
  18. ↑ Hsu, Stephen; Acharya, Sunil; Rafii, Abbas; New, Richard (25 April 2006). "Performance of a Time-of-Flight Range Camera for Intelligent Vehicle Safety Applications" (pdf). Advanced Microsystems for Automotive Applications 2006 (Springer): 205–219. doi:10.1007/3-540-33410-6_16. ISBN 978-3-540-33410-1. Archived from the original on 2006-12-06. Retrieved 2009-09-36. 
  19. ↑ Elkhalili, Omar; Schrey, Olaf M.; Ulfig, Wiebke; Brockherde, Werner; Hosticka, Bedrich J. (September 2006), "A 64x8 pixel 3-D CMOS time-of flight image sensor for car safety applications", European Solid State Circuits Conference 2006, pp. 568–571, doi:10.1109/ESSCIR.2006.307488, ISBN 978-1-4244-0302-8, retrieved 2010-03-05 
  20. ↑ Captain, Sean (2008-05-01). "Out of Control Gaming". PopSci.com. Popular Science. Retrieved 2009-06-15. 
  21. ↑ 21.0 21.1 Rubin, Peter (2013-05-21). "Exclusive First Look at Xbox One". Wired Magazine. Retrieved 2013-05-22. 
  22. ↑ 22.0 22.1 Sterling, Bruce (2013-06-04). "Augmented Reality: SoftKinetic 3D depth camera and Creative Senz3D Peripheral Camera for Intel devices". Wired Magazine. Retrieved 2013-07-02. 
  23. ↑ Lai, Richard. "PMD and Infineon to enable tiny integrated 3D depth cameras (hands-on)". Engadget. Retrieved 9 October 2013. 
  24. ↑ http://pewa.panasonic.com/components/built-in-sensors/3d-image-sensors/d-imager/
  25. ↑ http://www.softkinetic.com/products/depthsensecameras.aspx
  26. ↑ http://www.fotonic.com/content/Products/Default.aspx
  27. ↑ http://www.irisgmbh.de/products/irma-matrix/
  28. ↑ http://www.pmdtec.com/
  29. ↑ http://www.odos-imaging.com
  30. ↑ http://www.mesa-imaging.ch/prodview4k.php
  31. ↑ http://www.iee.lu/technologies
  32. ↑ http://www.brainvision.co.jp/xoops/modules/tinyd4/index.php?id=5
  33. ↑ http://www.tridicam.net/en/products/array-sensor
  34. ↑ "TowerJazz CIS Technology Selected by Canesta for Consumer 3-D Image Sensors". Business Wire. 21 June 2010. Retrieved 29 October 2013. "Canesta Inc. is using TowerJazz's CMOS image sensor (CIS) technology to manufacture its innovative CanestaVision 3-D image sensors." 

Further reading

  • Hansard, Miles; Lee, Seungkyu; Choi, Ouk; Horaud, Radu (2012). "Time-of-flight cameras: Principles, Methods and Applications". Springer Briefs in Computer Science. doi:10.1007/978-1-4471-4658-2. ISBN 978-1-4471-4657-5. "This book describes a variety of recent research into time-of-flight imaging: […] the underlying measurement principle […] the associated sources of error and ambiguity […] the geometric calibration of time-of-flight cameras, particularly when used in combination with ordinary color cameras […and] use time-of-flight data in conjunction with traditional stereo matching techniques. The five chapters, together, describe a complete depth and color 3D reconstruction pipeline." 

External links

This article is issued from Wikipedia. The text is available under the Creative Commons Attribution/Share Alike; additional terms may apply for the media files.