Cellphone cameras use AI and machine learning to sharpen up their image

Share

First there was the size of the camera lens. Then there were megapixel counts. Now, the one factor that is most likely to determine the quality of the photo you just took is the software and the artificial intelligence applied to it.

This third wave of imaging – also known as computational photography – is quickly eroding the conventional thinking of only a few years ago, that cellphones and their tiny lenses would never be able to produce results that rival full-size single lens reflex (SLR) cameras.

Newer smartphones, such as the Google Pixel 2 and the iPhone X, are proving the opposite.

"When I take a trip, I'll take a big camera with me and I'll also take my phone and I'll assiduously take out my large camera to take photos," says Marc Levoy, a principal engineer at Google. "But I'm always disappointed by the dynamic range, which is better on cellphones."

Google's first Pixel smartphone, released last year, introduced a high-dynamic range (HDR) mode that could instantaneously snap several photos at different lighting exposures. The device then stitches the multiple shots together into single photos that accurately convey both brightly lit and darker areas.

Many SLR cameras have similar HDR modes, but they can be slow and make errors when matching up multiple images.

"They really are far behind and need to catch up," says Dr. Levoy, who is also a professor emeritus of computer science and electrical engineering at Stanford University in California.

Reviewers gushed over the Pixel camera as a result. It garnered an 89 score on influential camera site DxOMark, the highest rating to date for a smartphone at the time.

The recently released Pixel 2 takes imaging a step further by utilizing machine learning, a term often used interchangeably with artificial intelligence because both involve algorithms that become smarter after crunching large amounts of data.

The Pixel 2 relies on algorithms that have studied millions of images, which gives its camera the ability to detect common features and techniques in photographs. It can automatically differentiate between foreground subjects and backgrounds, for example.

As with several competing phones, the Pixel 2 has a "portrait mode" that produces a sharp and crisp foreground subject with a slightly blurred background. A photographer using an SLR can achieve the same effect manually by focusing the camera's lens on the subject.

The Pixel 2 also differs from its competitors, including the iPhone X, in how it achieves this effect. Most current high-end phones have dual lenses that, in portrait mode, capture two separate images – background and foreground – that are then combined through software processing.

Google's device has just a single lens and relies instead on AI processing, which is yielding better results overall, according to DxOMark. The Pixel 2 has scored a 98, again the highest smartphone rating yet and one point better than the iPhone X.

Apple's device also uses machine learning, but the company says it doesn't draw processing power from the internet as many of its competitors do. Techniques such as AI-assisted facial recognition are done entirely on the iPhone in accordance with Apple's focus on privacy, according to a company blog post.

Rylo Inc., a San Francisco-based company started by former Apple and Instagram employees, is also relying on software processing to create better images. The company says its product, a $500 (U.S.) GoPro-sized camera, lets amateurs create professional-quality videos through fast and easy postproduction.

The Rylo camera shoots in 360 degrees and then effectively crops out the most desired portion of the entire image into a regular two-dimensional video. That means users can simultaneously eliminate camera shake and keep the focus on the subject, both of which are difficult for amateurs to achieve when spontaneously shooting.

"It's really hard to do because when you're capturing something, things are just happening and you don't know what's coming," says Alex Karpenko, Rylo co-founder and chief executive officer. "That never happens in [the film industry] because you have this entire crew that has planned everything ahead of time."

Full SLR cameras do still hold an advantage in some areas – especially when it comes to zoom, which only proper lenses are capable of doing properly so far.

But industry observers agree that traditional camera makers will need to focus less on hardware and speed up their adoption of computational photography, or risk ceding more of the market to software-oriented companies such as Google, Apple and Rylo.

"It's a major revolution," says Peter van Beek, professor in the Cheriton School of Computer Science at the University of Waterloo. "If the lens doesn't do it, you do it computationally."