
Astrophotography with Night Sight on Pixel Phones - JayXon
https://ai.googleblog.com/2019/11/astrophotography-with-night-sight-on.html
======
emptybits
This is amazing. Now just FYI, for those curious, some recent "old guard"
innovations by pro camera makers for astrophotography include:

1\. Leveraging the multi-axis image-stabilizing movement available to an in-
camera DSLR sensor with GPS for the purpose of tracking the sky during a long-
ish exposure to reduce star trails. Ricoh-Pentax Astrotracer.
[http://www.ricoh-imaging.co.jp/english/photo-life/astro/](http://www.ricoh-
imaging.co.jp/english/photo-life/astro/)

2\. Removing the camera's IR filter, allowing it to capture hydrogen-alpha
rays (656nm). This captures energy (image, color) not otherwise seen by normal
camera sensors. Canon EOS Ra.
[https://www.usa.canon.com/internet/portal/us/home/products/d...](https://www.usa.canon.com/internet/portal/us/home/products/details/cameras/eos-
dslr-and-mirrorless-cameras/mirrorless/eos-ra)

I know this only because I've been researching a DSLR/mirrorless camera
upgrade but also delaying it regularly while I'm reminded how excellent phone
cameras have become! Unless you're a pro, a pixel-peeper, an artist, or just
someone who simply enjoys the machinery and process.

~~~
dr_zoidberg
I didn't quite like the article. At the begining it shows a good picture of
the Milky Way and it says:

> The image has not been retouched or post-processed in any way.

Then the whole article describes how they've automated a full astroprocessing
pipeline inside the phone, that makes heavy postprocessing...

~~~
bla3
I think what "has not been retouched or post-processed in any way" is supposed
to convey is that no work is needed by the user of the camera app to get
pictures of this quality. There's lots of in-camera processing of course.
That's always been the case for digital cameras: sensors don't produce jpeg
files.

~~~
dr_zoidberg
That doesn't mean there wasn't heavy post-processing on the image. It simply
means the user didn't have to do it.

~~~
BeetleB
What people are trying to say is that if this is your definition of post
processing then there doesn't exist any digital camera system that produces
photos without post processing. Indeed such a camera cannot exist.

~~~
dr_zoidberg
I know, I work in digital image processing, and I'm an amateur
astrophotographer. Demosaicing is not on the same level as star registration,
stacking (posibly HDR processing in the middle), plus some more AI driven
"magic" in between.

That's what I'm referring to: for the shown image "without post-processing"
the phone has performed a lot more actions after demosaicing, that's why i say
it was heaviliy processed.

------
saiya-jin
As a full frame DSLR shooter (Nikon D750 + 20mm F1.8), this is wild
considering those crappy tiny sensors on phones. To get similar (albeit much,
much sharper) results, I have to lug around 2kg of camera and lens plus bulky
tripod.

Even with this, to get those dark dust clouds and stark colors some heavy
postprocessing is required (which I mostly don't do because I consider it too
much an alteration of original image, but it creates more interesting image).
Don't think for a second that those superb images you can see everywhere are
not literally over-painted in Photoshop (look at online tutorials on how to do
it if you don't believe me).

I guess to make things impressive, google guys went to some proper remote
desert far from any artificial light. And unless I missed something, they
still used some tripod. In european alps, this kind of result is practically
impossible - there is always some tiny village in every valley, and even if
not light pollution seeps from far. One night panorama I have has quite strong
glow coming from village of Chamonix some 15km far, that is on the other side
of massive Mont Blanc range [1]. Anything can be achieved if you start playing
a lot with Photoshop brushes, layers etc. but for me its one step too far.

Imagine what results can be had when such algorithms are paired to a full
frame (or bigger) sensor!

[1]
[https://www.flickr.com/photos/99251154@N04/22790364795/in/al...](https://www.flickr.com/photos/99251154@N04/22790364795/in/album-72157660842726331/)

~~~
lm28469
> To get similar (albeit much, much sharper) results

Tbh it's not that hard to get sharper results, this was shot on a 40 years old
camera, with a 50+ years old lens using a cheapo carbon fiber tripod ( 25+mph
wind that probably rocked my camera quite a bit)

[https://i.imgur.com/sdxyyEw.jpg](https://i.imgur.com/sdxyyEw.jpg)

> And unless I missed something, they still used some tripod.

They did: "Clearly, this cannot work with a handheld camera; the phone would
have to be placed on a tripod, a rock, or whatever else might be available to
hold the camera steady."

------
rbritton
Zoomed out, those photos aren’t terrible, but if you click to view a higher
resolution version, you can see the softness created by the noise removal
algorithm. Based on my experience, the image quality is comparable to DSLRs of
about ten years ago. Noise removal is a bandaid. In astrophotography, it will
virtually always look softer than reality actually is, and it will also have
removed actual stellar objects. There is no substitute for a better, lower
noise sensor, but you work with what you have.

Sensor size is inversely correlated with the noise level. Smaller sensors have
more noise where the least noise is on full frame (i.e., 35mm) or larger
sensors. Phones have small sensors.

~~~
baq
that's exactly what is amazing about this tech - 10 years ago you'd have to
carry a backpack, now the same quality pictures can be taken with a multi-
purpose device which fits in your pocket. nothing to complain about if you ask
me!

~~~
konart
I still get better photos with my almost 10 year old camera though.

UPD: oops, XT-1 is just 5 years old, actually. I guess I have to take my words
back, sort of.

Still, the point is the same - as long as you make a shot for your Instagram
account - your smartphone is alright. For anything bigger you still need a
camera with decent lens.

------
bla3
This is super cool tech. But I can't help thinking that this shows that the
Pixel camera lead must be someone with an engineering background who got nerd
sniped by this problem because it's cool and hard to do. Someone with a
product background would've focused on something less niche.

On the other hand, some people who like Google like it because it still
sometimes works on geeky, cool, fun stuff instead of being super product
focused.

~~~
tln
My impression is that night sight as a feature has been pretty impactful in
the market. As in, people buy them on this feature alone

~~~
bla3
Night sight for normal night photography, absolutely. Taking pictures in low-
light situations is common. Astrophotography on the other hand is much less
common.

------
JNRowe
I can't read about commodity hardware stargazing without my mind drifting to
2bit Astrophotography¹ which used the Gameboy Camera².

If you compare a paper star chart from the Gameboy era to Stellarium³ and a
Pixel, then it can only lead you to wonder where we will be in another 20
years.

1\. [http://pietrow.net/astrogb.html](http://pietrow.net/astrogb.html)

2\.
[https://en.m.wikipedia.org/wiki/Game_Boy_Camera](https://en.m.wikipedia.org/wiki/Game_Boy_Camera)

3\. [http://www.stellarium.org](http://www.stellarium.org)

------
cageface
I'd prefer to see Google focus on just making the Pixel line better everyday
phones than working on exotic stuff like this or the motion sensor they added
in the Pixel 4. Right now there's not much reason to choose a Pixel unless
stock Android is your top priority.

~~~
harrygeez
Having Chrome on my phone or choice of any alternative web browser is big
enough of a reason for me to pick Android. Besides, my experience with Android
in the last couple of years has been more solid and consistent than iOS, even
if the UI and animations are less polished.

~~~
cageface
I agree and I prefer the UX on Android in a lot of respects too. I just don’t
think the Pixels have been a good value compared to other Android phones
lately. They either need to be cheaper or better than they are now.

------
habosa
The photos are gorgeous and I would definitely use this mode.

Astrophotography always kind of rubs me the wrong way though because that's
not how it looks. Even if you go out to somewhere that's really dark, like a
large National Park, and wait for a clear night it's never going to look in
your eye like it does on Instagram. Don't get me wrong, what you do see is
absolutely magnificent, it's just not what's in those pictures.

Seeing the galaxy with your own eyes is one of the most majestic things you'll
ever witness. It's something that has inspired spontaneous prayer throughout
history. It doesn't really need a filter.

------
jakecopp
If all the advancements in smartphone photography is in software, why aren't
DSLR/mirrorless manufactures doing it too?

I don't want my camera to have a touchscreen/social media/wifi but it'd be
cool if Adobe Camera Raw/some alternative could do this stuff!

~~~
ygra
IMHO they've basically missed an opportunity here for many years. They'd be in
a perfect position to offer those things, combined with a much better/larger
sensor, which enables even better images. On smartphones it's a matter of
necessity, as the sensor is (fairly) crappy in comparison, but on a DSLR it
could still be a benefit. Personally I'd be perfectly happy to get a pre-
processed DNG from the camera instead of having to do this afterwards. And
then give me the raw files to do it manually as well.

Perhaps they're trying not to cannibalize their lower market segments or think
that professionals would never use those things (on which they _might_ be
correct). But I can definitely see that computational photography beyond
raw->JPEG conversion with a color profile could have its place in a DSLR.

~~~
SiempreViernes
While it would be nice to load some post-processing script onto the camera and
get all that done by the camera without having to bother with it yourself, all
that processing _will_ cost battery life, as well as response time. Having to
charge your DSLR every day is a huge cost in convenience, and making a
_dedicated_ imaging machine unresponsive would be terrible.

Additionally, you are always going to move your pictures out of the camera for
proper viewing, so it doesn't make a lot of sense to provide the best possible
picture already in the camera.

I don't know if the software they bundle with the camera is any good at this
computational photography though (I've never checked to see if there are linux
versions), but it better be if it indeed helps image quality.

~~~
ygra
Yeah, it's definitely nice to get ~1000 images per battery charge. But when
uploading to the PC infrequently you'll then have to sort photos into groups
to process individually (something I already hate with panoramas). Personally
it's something I'd rather not do, at the expense of only getting ~200 photos
per charge. It's also not necessary to compromise, as those special modes
would be, well, special modes. So to preserve battery I could just as well
shoot raw as normal.

------
PopePompus
The photos taken in this mode are amazing. But I think they are improved a lot
if a vignetting correction is applied. Without that correction, the sky
brightness is much greater near the center of the field than near the edges.

------
kohtatsu
Any recommendations for tripods/cases?

iPhone 11 Pro here.

------
imvetri
What are the possibilities that this AI camera app is not faking the image.

Here is why I ask \- It has gyro - so knows whether we are pointing at sky or
not. \- AI checks whether its a clear sky - if yes - post fake image. If no -
Dont risk getting caught. \- Time + geo spacing - Gives the angle, position of
camera relative to the space above us.

~~~
PopePompus
There is no chance this is being faked. I've been playing with the
astrophotography mode on a Pixel 4 XL in a remote country location with no
cell phone connectivity. A single 4 minute exposure is able to record stars
down to about magnitude 9.5. A single exposure can detect the Crab Nebula, or
the two brightest satellite galaxies of the Andromeda Galaxy (M 32 and NGC
205). To fake such results without internet connectivity would mean that the
camera software would have to have an internal catalog of about 250,000 stars,
with accurate colors and coordinates. It would need the size and shape of
nebulae, the contours and brightness of the Milky Way, etc. When I add
multiple frames, I see fainter stars appear, as they should, so the internal
catalog would really need millions of stars, most of which would not show up
unless the user carefully aligned multiple individual image and summed them.
This is not being faked.

~~~
Jyaif
You do realize that with 2 floats to store the position of a star on a sphere,
that's like 8 MB per million of star? With compression you can almost
certainly get away with something like 2 bytes per stars.

And an other solution is to simply store a bitmap of the sphere around earth.

~~~
PopePompus
It also shows the positions of the planets correctly, so it would need to have
an ephemeris, or planetary orbital elements. Meteors and satellites show
trails across the star field. At some point simulating all of this would be
more complicated than really doing it.

------
ganitarashid
As long as you manually control the exposure and exposure time, the same can
be achieved with an iPhone camera. This is not specific to Pixel.

~~~
oceanofsolaris
They mention a couple of features that are not just "long exposure" in the
post:

* Compensating for moving stars

* "Live viewfinder" during exposure

* Selectively darkening the sky

* Dark current compensation (though that is probably needed for all long-exposure photography...still, not a simple "more exposure" feature)

~~~
ganitarashid
All easy software implementations. This is not a hardware story.

~~~
jeen02
If they're so easy, why haven't Apple implemented them?

~~~
piyush_soni
Or at least someone would have come up with an app already, had it been that
easy.

