Written in Lytro Ten Years - The shortcomings, Advantages, Applications and prospects of Light field cameras (Part 1)

It's been nearly a decade since Lytro introduced its first generation of light-field cameras to the market in 2011. This so-called "one shot, multiple focus" imaging equipment, at the time of its first commercial release, in the photography industry, technology and media have caused a stir. Compared with traditional SLR cameras, its main selling point is that it does not need to focus and look for photography angles, and you can get photos with different focal depths and angles after a single shot. Ren Ng, inventor of the light-field camera and founder of Lytro, received an ACM Dissertation Award for his doctoral thesis. He also received "MIT Tech Review's TR35","Fast Company's 100 Most Creative People in Business", and "Silicon Valley Journal's 40" under 40" and so on. The light field camera can be described as a moment of boundless beauty, while being hailed as a subversive technological invention, some people even regard it as the historical terminator of the SLR camera.

Ten years later, the company responsible for developing and promoting the product, Lytro, closed down in 2018, and the company's CEO, Ren Ng, who also invented the camera, chose to return to academia as a professor at UC Berkeley. The camera experienced a severe high opening and low running, as if it disappeared in the field of view of consumers.

Although in the To C end, the performance of the light field camera can be described as terrible, and the feedback of users is also a tide of negative comments, but in the To B end and the academic community, the light field camera still continues to have a relatively strong vitality, not only academic articles are published one after another, expanding the scope of application of the light field camera. At the same time, a small number of technological unicorns at home and abroad will also land the technology in some application fields (such as Raytrix in Germany and Shanghai Yimu Technology in China, they can be described as the two leaders in the field of optical field imaging in industrial applications).

There are even technological inventions based on light field imaging that have won national major scientific and technological progress awards, and can also be seen in the world's top academic journals (for example, Dai Qionghai's research group at Tsinghua Graduate School in Shenzhen has had a greater impact on the biomedical microscopy application of light field imaging).

This seems to be a technology, undergoing some kind of rise and fall change, in the history of technology there is no shortage of a certain technology rise and fall alternating phenomenon, driving people in the academic community and the industry to rush forward, break up, and continue to participate in or leave a technical field.

In the case of light-field cameras, this paper attempts to answer a reflective question about this phenomenon from a purely technical point of view: What are the advantages and disadvantages of light-field cameras? What causes it to be abandoned by consumers while continuing its life in the industrial world? What is the status quo of landing applications? This article is not only to ask yourself a self-answer, but also to throw a brick to introduce jade, I hope that you engaged in this industry, can publish their own opinions.

一、Insufficient

Looking back on the shortcomings of the equipment is also a review and explanation of the gradual abandonment of the light-field camera in the To B market. Because of its own hardware structure, the optical field camera cannot avoid three major congenital defects.

1. Low spatial resolution

Light field camera structure (from "Bionic Vision-based single-camera Light Field Imaging and 3-3 dimensional Direct conversion Basis")

In order to collect information from different angles, a microlens array is placed in front of the CCD. However, the total amount of information that can be collected by the camera is limited by the size and number of CCD pixels on the imaging chip. The relationship between Angle information and spatial information can be simply described as:

As I pointed out in the other answer, the size of the light field image in a single viewing Angle is equal to the number of microlenses in the microlens array, and since the number of pixels in a single microlens is between 100 and 200, the number of microlenses is equivalent to 100 times the number of pixels in the original CCD. For a 10 million pixel light field camera, after compromise compression of Angle information, the number of effective pixels obtained in a single Angle is only 100,000 pixels! Compared with the tens of millions of pixels of traditional SLR cameras, this order of spatial resolution is like a toy for consumers who use photography. This is also the biggest factor in the disappearance of light-field cameras in the consumer market. Even if Lytro subsequently launched a 2.0 generation of optical field cameras to further compress the angular resolution to improve the spatial resolution, the above problems have not been completely solved. However, this defect should be treated dialectically, and the relative choice of Angle information and spatial information depends on the application scenario.

2. Short baseline
The length of the baseline problem is not sensitive to consumers, but it is an extremely important parameter in the field of macro imaging and measurement, such as aerial photographic mapping, binocular stereo vision, satellite remote sensing imaging.

Baseline and parallax relationship diagram

In multiocular vision, the perception of depth information by the image recipient (human, machine) is derived from the parallax of the object on different cameras (eyes), and the magnitude of the parallax changes with the distance of the object from the camera. The distance length between cameras is called the baseline length, and under the same distance condition, the larger the baseline length, the greater the parallax and the higher the depth resolution ability.

For binocular vision, its baseline length can be expressed by the spacing between two cameras arranged roughly in parallel, while for light-field cameras, how is its baseline length calculated? As shown in the figure below, because each microlens records part of the information of the incident point light source, at this time you treat each microlens as a small camera, then you can intuitively get the answer, because of the existence of multiple "small cameras", the light field camera has multiple baselines, but the maximum baseline length of the light field camera is the distance between the two microlenses that are lit furthest apart. But this baseline length, there are two characteristics: First, the baseline length changes with the defocusing distance of the point light source. When the defocusing plane is near, only a few microlenses are lit, so the maximum baseline length is shorter. However, when the defocusing plane is far away, more microlenses are lit, and the maximum baseline length is larger (this is also the reason why the light-field camera is near the focal plane and has the worst depth-of-field resolution). However, the baseline length of the microlens cannot exceed the CCD chip size itself.

This gap, and binocular vision contrast, is especially obvious, such as the baseline length of the light field on the right side of the above figure, roughly around the diameter of 2 to 3 microlenses. Because the baseline of the light field camera is too short, the depth resolution will be greatly limited when the measured object is far away.

3. The camera's dynamic range HDR is compressed
Using the analysis case of the above point light source, it can be found that when the point light source is near the focal plane, only a few microlenses are lit, and there must be more light energy concentrated on a few CCDS. When the point light source defocuses the plane courtyard, multiple microlenses are lit, that is, the energy distribution of the point light source is more dispersed. In this condition, in order to allow the camera CCD to be exposed or too dark, the brightness of the photographed object can only be in a relatively small dynamic range, so that the object will not be overexposed when it is close to the focus, and will not be too dark when it is far from the focus.

The energy distribution of point light source at different distance from the focal plane

The above three problems seriously limit the application of light field cameras, and are also the main reasons why light field cameras have not been accepted by consumers in the past decade. However, there are strengths and weaknesses, in the To B end and the school, the light field camera has many unique advantages, and can complete the tasks that traditional imaging equipment can not. The above three deficiencies have been overcome and solved to a certain extent in these application fields. In addition, from a dynamic perspective, with the reduction of pixel size, the increase in the number of pixels on the chip, the emergence of higher dynamic range chips (and even some chips claim to have developed technology that will never be revealed), the above problem is expected to be fundamentally solved.

Technology booms and busts, as they often do, turn from village to village, attracting the "Boom and Bust" of each decade.