System can apply a range of styles in real-time, so that the viewfinder displays the enhanced image.
The data captured by today’s digital cameras is often treated as the raw material of a final image. Before uploading pictures to social networking sites, even casual cellphone photographers might spend a minute or two balancing color and tuning contrast, with one of the many popular image-processing programs now available.
This week at Siggraph, the premier digital graphics conference, researchers from MIT’s Computer Science and Artificial Intelligence Laboratory and Google are presenting a new system that can automatically retouch images in the style of a professional photographer. It’s so energy-efficient, however, that it can run on a cellphone, and it’s so fast that it can display retouched images in real-time, so that the photographer can see the final version of the image while still framing the shot.
The same system can also speed up existing image-processing algorithms. In tests involving a new Google algorithm for producing high-dynamic-range images, which capture subtleties of color lost in standard digital images, the new system produced results that were visually indistinguishable from those of the algorithm in about one-tenth the time — again, fast enough for real-time display.
The system is a machine-learning system, meaning that it learns to perform tasks by analyzing training data; in this case, for each new task it learned, it was trained on thousands of pairs of images, raw and retouched.
The work builds on an earlier project from the MIT researchers, in which a cellphone would send a low-resolution version of an image to a web server. The server would send back a “transform recipe” that could be used to retouch the high-resolution version of the image on the phone, reducing bandwidth consumption.
“Google heard about the work I’d done on the transform recipe,” says Michaël Gharbi, an MIT graduate student in electrical engineering and computer science and first author on both papers. “They themselves did a follow-up on that, so we met and merged the two approaches. The idea was to do everything we were doing before but, instead of having to process everything on the cloud, to learn it. And the first goal of learning it was to speed it up.”
In the new work, the bulk of the image processing is performed on a low-resolution image, which drastically reduces time and energy consumption. But this introduces a new difficulty, because the color values of the individual pixels in the high-res image have to be inferred from the much coarser output of the machine-learning system.
In the past, researchers have attempted to use machine learning to learn how to “upsample” a low-res image, or increase its resolution by guessing the values of the omitted pixels. During training, the input to the system is a low-res image, and the output is a high-res image. But this doesn’t work well in practice; the low-res image just leaves out too much data.
Gharbi and his colleagues — MIT professor of electrical engineering and computer science Frédo Durand and Jiawen Chen, Jon Barron, and Sam Hasinoff of Google — address this problem with two clever tricks. The first is that the output of their machine-learning system is not an image; rather, it’s a set of simple formulae for modifying the colors of image pixels. During training, the performance of the system is judged according to how well the output formulae, when applied to the original image, approximate the retouched version.
The second trick is a technique for determining how to apply those formulae to individual pixels in the high-res image. The output of the researchers’ system is a three-dimensional grid, 16 by 16 by 8. The 16-by-16 faces of the grid correspond to pixel locations in the source image; the eight layers stacked on top of them correspond to different pixel intensities. Each cell of the grid contains formulae that determine modifications of the color values of the source images.
That means that each cell of one of the grid’s 16-by-16 faces has to stand in for thousands of pixels in the high-res image. But suppose that each set of formulae corresponds to a single location at the center of its cell. Then any given high-res pixel falls within a square defined by four sets of formulae.
Roughly speaking, the modification of that pixel’s color value is a combination of the formulae at the square’s corners, weighted according to distance. A similar weighting occurs in the third dimension of the grid, the one corresponding to pixel intensity.
The researchers trained their system on a data set created by Durand’s group and Adobe Systems, the creators of Photoshop. The data set includes 5,000 images, each retouched by five different photographers. They also trained their system on thousands of pairs of images produced by the application of particular image-processing algorithms, such as the one for creating high-dynamic-range (HDR) images. The software for performing each modification takes up about as much space in memory as a single digital photo, so in principle, a cellphone could be equipped to process images in a range of styles.
Finally, the researchers compared their system’s performance to that of a machine-learning system that processed images at full resolution rather than low resolution. During processing, the full-res version needed about 12 gigabytes of memory to execute its operations; the researchers’ version needed about 100 megabytes, or one-hundredth as much. The full-resolution version of the HDR system took about 10 times as long to produce an image as the original algorithm, or 100 times as long as the researchers’ system.
“This technology has the potential to be very useful for real-time image enhancement on mobile platforms,” says Barron. “Using machine learning for computational photography is an exciting prospect but is limited by the severe computational and power constraints of mobile phones. This paper may provide us with a way to sidestep these issues and produce new, compelling, real-time photographic experiences without draining your battery or giving you a laggy viewfinder experience.”
Learn more: Automatic image retouching on your phone
The Latest on: Image enhancement
- ANNKE Unveils New NightChroma™ Cameras - World's First ACE True Full Color Night Vision Smart Security Cameras Globallyon November 27, 2020 at 11:48 am
ANNKE, created for security, today unveils world's first ACE true full color night vision security cameras — NC400 and NCA500, ...
- Forget Black Friday TV deals – these home theater projectors start at $500on November 26, 2020 at 12:15 pm
crisp images — with Image Enhancement and Frame Interpolation Built-in Android TV (2) — latest Android TV interface with a simple-to-use remote – including voice search with built-in Google ...
- Hubble Captures Black Hole's “Strange Rays” Lighting Up A Recently Merged Galaxyon November 25, 2020 at 9:38 am
Rays of light shooting out from the supermassive black hole at the center of a galaxy are lighting up surrounding dust even as we can't see the source. The ...
- Sedona Camera Club offers two-part webinar ‘Plugins for Image Enhancement’ and ‘The Art of Travel Photography’on November 24, 2020 at 9:48 am
Photographic artist JR Schnelzer will present to the Sedona Camera Club starting at 6 p.m. on Monday, Nov. 30. Due ...
- Epson Expands Home Cinema Projector Lineon November 9, 2020 at 3:00 am
Additionally, the 2250 and 2200 both have image enhancement and frame interpolation, and both projectors feature Bluetooth connectivity. The Home Cinema 1080 and 880 both feature 1080p resolution ...
- IMAX's Q3 Loss Wider Than Expected, Revenues Decline Y/Yon October 30, 2020 at 10:40 am
Category-wise, technology sales, Image Enhancement & Maintenance Services, Technology Rentals and financial income were down 27.5%, 67%, 74.6% and 14.2% year over year to $15.8 million ...
- Apollo 13: Enhanced images reveal life on stricken spacecrafton August 7, 2020 at 11:13 pm
image captionApollo 13's commander Jim Lovell selects music on a portable tape player while Jack Swigert naps off to the right Image enhancement techniques have been used to reveal life aboard ...
- Chapter 6: Image Enhancement in the Spatial Domainon February 21, 2018 at 8:43 am
Image enhancement is the processing of images to improve their appearance to human viewers, in terms of better contrast and visibility of features of interest, or to enhance their performance in ...
- Image Enhancement GUI Using MATLABon November 4, 2017 at 12:02 pm
In this paper, the authors present an overview of image enhancement processing techniques in spatial domain and developed its Graphical User Interface (GUI) in MATLAB. This paper describes the ...
- image enhancement module pythonon September 28, 2015 at 1:08 pm
I'm trying to implement image processing s/w that requires me to implement an image enhancement module that performs various operations like Color, Contrast, Brightness, Sharpness manipulation on the ...
via Google News and Bing News