Extreme imaging using cell phones: SeeInTheDark [pdf]

Extreme imaging using cell phones: SeeInTheDark [pdf](graphics.stanford.edu)

198 points by jwise0 9 years ago | 23 comments

notlisted 9 years ago |

Nice PDF. Some cool new stuff, smart decisions & remarks (noise, hotpixels, matching frames, etc).

If you have an iOS device, check out the app Average Cam Pro[1]. I've had it on my iPad for several years now. It does an awesome job taking noise-free, detailed pictures (as long as you don't hold it in your hand) by taking up to 100 pictures and merging them. You can adjust exposure etc. after the fact. Great stuff.

I have yet to find a good replacement for my Android phone. The closest thing to [1] is Multiple Exposure Camera [2] which has a terrible interface and no exposure setting, but some other nifty settings to remove moving objects etc. (note: just don't use the 0s interval or it crashes)

[1] https://itunes.apple.com/us/app/average-camera-pro/id4155778...

[2] https://play.google.com/store/apps/details?id=com.conslazy.m...

ricw 9 years ago | |

I just installed average camera pro. Very blurry. Not useful for indoor photography. Does not allow for adjustment of the shutter speed. Don't buy.

It's Nothing like what the paper describes.

notlisted 9 years ago | | |

Operator error. Like I said, it's not for handheld use (the only way to get blurry shots).

Correct way:

- Settings: Set the number of shots to 32 or higher. 1s interval. 3s pre-start timer

- Point at scene. Lean it against an object (don't touch/move) or mount on a tripod.

- Select exposure/focus/whitebalance with longpress (I select a light part of the scene, exposure/gain can be adjusted upward afterwards).

- Lock exposure/focus/whitebalance (could be an L in the middle, or text on your screen left, boldface is locked in)

- Shoot.

- After it's done (don't touch!) adjust the exposure if needed.

====

If you want even higher quality images, you can subtract banding noise (also mentioned in the PDF by the way).

In order to get the banding noise image for your camera:

- set shots to 128, delay 5 seconds

- point lens at light source (e.g. on table, don't move)

- hit shoot, put piece of white paper over the lens

- wait ~2mins (till it stops shooting).

- Select N in middle, "save image as noise fingerprint".

Then take another shot of a scene. Once final pic shows, hit N, now say "remove banding noise".

notlisted 9 years ago | | |

@ricw - here, a very quick and dirty comparison, iPad3, dirty keyboard, watch, focus/exposure locked on rim of the watch.

https://goo.gl/photos/9FM2c9JZRcDitPdd6

Watch full screen for best comparison. Taken in dark office, with just a desk light.

ChuckMcM 9 years ago |

That was a great read, back in the early 80's at the Image Processing Lab at USC there was a project to turn, what was then, a new "CCD" imager into something that could collect images that were useful for image analysis. At the time these "TV" cameras were equivalent to a really low end web camera. What we came up with was basically the first half of this talk, taking frame after frame and integrating it into the frame buffer with a filter function. Sad that I can't find the technical reports in Google Books.

andmarios 9 years ago |

Very impressive! I have an app on my android that follows the same basic idea (accumulate frames) but doesn't come anywhere near this.

Unfortunately the author, in a reply to a youtube comment, stated that this is work he did for Google and doesn't know if the company has any plans for this.

puzzle 9 years ago | |

He's Marc Levoy and he knows Google's plans very well; he just can't talk about them on YouTube, since it's the stuff that gets announced on stage at I/O and similar events.

nullc 9 years ago |

There is an old free software package ALE ( https://unix4lyfe.org/ale/ ) that does some of the processing like this. It was much easier to get working if you had linear light images... and registration of really noisy images was quite hard on it.

nmstoker 9 years ago |

Here's a (v short) video by the paper's author showing the app in action with there manuscript that features early on in the paper: https://youtu.be/S7lbnMd56Ys

br1n0 9 years ago |

In php is quite simple to merge images by pixel, I also tried to delete moving object https://github.com/br1n0/phpPhotosMerger

tambourine_man 9 years ago | |

Nice job

ricw 9 years ago |

Is this what makes the google pixel phone camera so outstanding? it would explain why a technically inferior camera (slower aperture, no optical image stabilisation) outperforms both apple and Samsung flagship phones.

mparlane 9 years ago |

Mirror [85 MB]: https://dl.dropboxusercontent.com/u/9910153/seeinthedark-pub...

jjcm 9 years ago | |

Another mirror in case it goes over the dropbox limit: http://files.jjcm.org/seeinthedark-public-15sep16.key.pdf

mparlane 9 years ago | | |

Thanks, it did :)

mparlane 9 years ago | |

Dropbox suspended my public folder due to large bandwidth to the mirror.

I think I can post this instead as it doesn't use my Public folder: https://www.dropbox.com/s/ixruqswy86fwr8t/seeinthedark-publi...

trhway 9 years ago |

Hubble - does it take one long exposure or similarly multiple short ones?

dandelany 9 years ago | |

A bit of both. Hubble can have single exposures that are hundreds of seconds long - but remember it's in low orbit, so eventually the subject will fall beneath the horizon. Most Hubble observations stitch together exposures from multiple orbits. Ultra Deep Field is an extreme example which comprises ~a million seconds of exposure over 400 orbits: https://en.m.wikipedia.org/wiki/Hubble_Ultra-Deep_Field

xioxox 9 years ago | | |

In astronomy you need multiple exposures to get rid of cosmic rays, which appear as bright lines or dots. A median filter can be applied to the exposures. HST uses tools called drizzlepac and multidrizzle to combine multiple dithered images to better make use of the telescope's native resolution which would otherwise be degraded by the detector.

Aoyagi 9 years ago |

>Need long exposure, but cell phones have no shutters.

Mine have... 1020 and 808.

dsjoerg 9 years ago |

excellent! now someone make an app out of this.

hughes 9 years ago |

Sounds interesting, too bad the server is dead