一、Introduction
It is undeniable that the current mainstream end-to-end optical algorithm joint design is mostly from foreign research teams, and China still needs to strengthen in related fields. However, the domestic related imaging industry is developed, especially in mobile phones, industry, vehicle and other fields, the development of new technologies is often easy to get rapid industrial application, and reverse promote academic progress, forming a virtuous cycle.
Based on this, the research team of Yimu Technology, Google, Snap, Tongji University, King Abdul University of Science and Technology and the Chinese University of Hong Kong has published a new paper titled "Latest Review: In "Advances in Computational Imaging", new design methods, algorithms and application scenarios of computational imaging are taken as the main line, and the main progress in this field is reviewed by synthesizing domestic and foreign literature and relevant reports. The research directions include end-to-end optical algorithm joint design, high dynamic range imaging, light field imaging, spectral imaging, lensless imaging, low illumination imaging, three-dimensional imaging, computational photography, etc. This paper focuses on the development status, frontier trends, hot issues and trends in the field of computational imaging.
二、Content introduction
This paper comprehensively describes the current development status, frontier trends, hot issues, development trends and application guidance of computational imaging, including end-to-end optical algorithm joint design, high dynamic range imaging, light field imaging, spectral imaging, lensless imaging, polarization imaging, low illumination imaging, active 3D imaging, computational photography and other specific examples.
1.High dynamic range imaging
In computational graphics and photography, high dynamic range imaging (HDR) is a technique used to achieve a larger exposure dynamic range (the ratio of the brightest and darkest details) than normal digital imaging techniques. High dynamic range imaging aims to obtain brighter and darker details, thus bringing richer information and more shocking visual impact, which is not only one of the core competitiveness of the current mobile phone camera, but also the basic requirements of industrial and car cameras.
2.Light field imaging
light field imaging (LFI) can record the spatial position and Angle information of light simultaneously, which is a new method of three-dimensional measurement. After recent years of development, it has gradually become an emerging non-contact measurement technology. Since the invention of photography, image capture has involved obtaining information in the two-dimensional projection of the scene. However, the light field not only provides a two-dimensional projection, but also adds another dimension, namely the Angle of light reaching that projection. The light field holds information about the orientation of the light array and the two-dimensional projection of the scene, and can achieve different functions. For example, the projection can be moved to a different focal length, which gives the user the freedom to refocus the image after acquisition. You can also change the perspective of the captured scene. At present, it has been gradually applied in industry, virtual reality, life science and 3D flow testing and other fields to help quickly obtain real light field information and complex 3D spatial information.
3.Spectral imaging
spectrum imaging (spectrum imaging). Spectral imaging is usually divided into scanning type and snapshot type, in which snapshot type because it only needs a single exposure to obtain the entire data cube information, has more potential and prospect in application, so the current research on spectral imaging in computer vision and graphics is also mainly focused on snapshot spectral imaging technology. According to different realization methods, snapshot spectral imaging can be divided into sub-aperture spectral imaging, sub-image plane spectral imaging, aperture/image plane coding spectral imaging. As a powerful tool for scientific research and engineering applications, spectral imaging has been widely used in military, industrial, civil and many other fields, and plays an important role in promoting social and economic development and ensuring national security. For example, spectral imaging has a good recognition effect on river, sand, vegetation, rocks and minerals, so it has important applications in precision agriculture, environmental monitoring, resource exploration, food safety and many other aspects. In particular, spectral imaging is also expected to be used in terminals such as mobile phones and self-driving cars. At present, spectral imaging has become one of the hot research directions in computer vision and graphics.
4.Lensless imaging
lensless imaging technology provides a new way to further compress the size of imaging systems (Boominathan et al., 2022). Lentless imaging abandoned the point-to-point mapping mode in the traditional lens, but projected the point of the object space as a special point diffusion function of the image space. Different object points were superimposed on the image plane, forming a kind of original data that the human eye could not recognize, but the calculation algorithm could recover. This lensless imaging method encodes image information on optical hardware and decodes it in a computational algorithm, thus forming a joint optical and algorithmic design. Therefore, it has great development potential in wearable cameras, portable microscopes, endoscopes, Internet of Things and other applications. In addition, its unique optical encryption function, which can effectively protect sensitive biometric characteristics in the target, is also of great significance in artificial intelligence imaging for privacy protection.
5.Low light imaging
low light imaging is also a research hotspot in computational photography. Mobile phone photography has become one of the most commonly used ways for people to record life, the camera function of the mobile phone is also the focus of each conference, and the night scene mode has become the technical commanding heights for major mobile phone manufacturers. The difference between the cameras of different mobile phones is not obvious in the strong light environment during the day, but the difference is obvious in the low light condition at night. The reason is that imaging relies on the lens to collect the photons emitted by the object, and the sensor will have inevitable noise from a series of processes such as photoelectric conversion, gain and analog-to-digital conversion. During the day, the light is sufficient, the signal-to-noise ratio of the signal is high, and the imaging quality is high. At night, the light is weak, the signal-to-noise ratio of the signal drops several orders of magnitude, and the imaging quality is low. Some mobile phones are equipped with night view mode using computational photography algorithms, such as noise reduction based on single frame, multi-frame, RYYB array, etc., which effectively improves the quality of photos. But there is still plenty of room for improvement. According to the input classification, low light imaging can be divided into single frame input, burst imaging, flash assisted shooting and sensor technology.
6.Active 3D imaging
Active 3D imaging aims to obtain point clouds of objects or scenes, while passive methods are represented by binocular stereo matching, but it is difficult to solve the depth of untextured areas and areas with repeated textures. Active light methods are generally more robust, can work in the dark, and can obtain dense, accurate point clouds. Active light methods can be divided into light based linear propagation such as structured light, light based speed of light such as Time-of-fligt (TOF), including continuous wave TOF (iTOF) and direct TOF (dTOF), and light based wave properties such as interferometers, according to the nature of the light used. The first two methods of active 3D imaging have been widely used in People's Daily life.
三、Summary
The end-to-end optical algorithm joint design aims to break the barriers between optics, sensors, image post-processing algorithms and display processing in traditional imaging systems, reduce the dependence of each link on human experience, and provide a new solution for many scenes.
Looking to the future, although end-to-end imaging technology is still in its early stages, it has become the focus of competition in major related industries, and in the short term, it can be applied in relevant industries in the next three years. In the long run, the overall trend is not only the joint design optimization of optics, sensors, algorithms and processors, but also the development of integration, especially the integration and optical cooperation of the sensor itself to the ISP, which will greatly reduce the production cost of the entire optical and camera module industry and reduce the dependence of downstream manufacturers on high-performance processors and improve the degree of freedom. It is expected to reshuffle the overall imaging industry chain and break through cost and function bottlenecks.
For more details on the paper, click on the Latest Review:Advances in Computational Imaging Frontiers
About VOMMA
VOMMA (Shanghai) was established in 2019. Through long-term research and development, we have developed a new generation of three-dimensional detection technology for light field. Through bionic insect compound eye imaging, tens of thousands to tens of millions of tiny compound eye sensors are precisely packaged into a camera, realizing "single camera, single shot, three-dimensional imaging", and solving the problems of "transparent, thin, and micro" three-dimensional size and rapid and precise defect detection.
VOMMA is the first high-tech enterprise in China to fully grasp the full link core technologies of optical design, micro and nano processing, packaging and manufacturing, algorithm software, etc., with industrial scale and standardized mass production capabilities of optical field cameras, and the precision and speed of optical field systems are in the forefront of the world. The main products are used in pan-semiconductor, lithium battery, smart wear, 3C, micro and nano three-dimensional detection, is the world's leading supplier of light field cameras.