
Video-to-Video Synthesis [video] - nakami
https://www.youtube.com/watch?v=S1OwOd-war8
======
John_KZ
It's nice but I wonder why there's still nobody that tried to create a
spatially coherent 3D space from the input before re-synthesizing the output.
It's definitely possible and lots of work on this was done in late 2017. There
should be more papers on creating and rendering 3D representations by now.

~~~
rzzzt
I vaguely remember a demo showing a 3D model of a winter trail, which was
recreated (and textured) from a sequence of photos. It was done with
Photosynth, I believe:
[https://en.wikipedia.org/wiki/Photosynth](https://en.wikipedia.org/wiki/Photosynth)

~~~
piceas
I didn't find the winter trail video but stumbled upon some interesting links:

2018 3D Scanning: A Comprehensive Survey
[https://arxiv.org/abs/1801.08863](https://arxiv.org/abs/1801.08863)

2012-2014ish BigSFM: Reconstructing the World from Internet Photos
[http://www.cs.cornell.edu/projects/bigsfm/](http://www.cs.cornell.edu/projects/bigsfm/)

David J. Crandall
[https://www.cs.indiana.edu/~djcran/#research](https://www.cs.indiana.edu/~djcran/#research)

Andrew Owens [http://andrewowens.com/](http://andrewowens.com/)

Also check out his on/offscreen audio source separation work. It's pretty
neat. (2018) Audio-Visual Scene Analysis with Self-Supervised Multisensory
Features
[http://andrewowens.com/multisensory/](http://andrewowens.com/multisensory/)

Noah Snavely
[http://www.cs.cornell.edu/~snavely/](http://www.cs.cornell.edu/~snavely/)

(2018) MegaDepth: Learning Single-View Depth Prediction from Internet Photos
[http://www.cs.cornell.edu/projects/megadepth/](http://www.cs.cornell.edu/projects/megadepth/)
[https://www.youtube.com/watch?v=tGbMWAFMMBQ](https://www.youtube.com/watch?v=tGbMWAFMMBQ)

With live demo page :)
[http://megadepthdemo.pythonanywhere.com/](http://megadepthdemo.pythonanywhere.com/)

------
pwaai
Does this mean that in the future we could literally be able to play FMV
games?

Basically, an AI that has been trained on every digital film there is which in
turn also generates unique permutations to further increase the training
data....

Will be able to conjure up things like Breaking Bad Season 8: 'Lil Heisenberg
and pretty much even helmet cam videos from war zones in Donetsk that you
could control in real time....

Singularity indeed.

~~~
abraham_lincoln
I want remakes of movies where characters can be swapped out from unrelated
movies.

Basically, a simulation, with the plot as objectives.

------
person_of_color
Interesting but how do we know the examples are not cherry-picked?

~~~
gwern
Well, first, Nvidia has a good track record. This is not their first video-to-
video paper, and they've also released impressive things like ProGAN; they
release not just the source code, but trained models as well. People have
trained their own models (slowly) and poked at the released models. The
average result might not be _quite_ as good, but they'll be close.

Second, in this case, they've also released source code and the trained
models. The trained models aren't in the Github repo, but hosted on Google
Drive and they provide scripts for downloading them:
[https://github.com/NVIDIA/vid2vid/tree/master/scripts](https://github.com/NVIDIA/vid2vid/tree/master/scripts)

If you are doubtful, simply install Pytorch, download the models, and give the
video generation a try. (You don't need a GPU, even, since for generating some
samples once, a CPU will be acceptably fast.)

~~~
screye
Wait, this algorithm works at real-time on consumer grade hardware ? I am
impressed.

I thought a forward pass for the model would be a lot more expensive than
that.

~~~
gwern
I never said it was realtime. I said it was acceptably fast on CPU ie. it'd
take a few minutes/hours to generate some samples and convince yourself that
they did not cherrypick a tiny handful of good quality samples out of a
universe of lousy ones. I would very roughly guesstimate their 2000px model
probably would take ~0.5s/frame on a GPU, so multiply by the usual >10x CPU
slowdown, you can generate a 30fps 30s snippet in maybe an hour or so.

------
cjdell
Wonder if this can be used to do video upscaling on steroids. Would love to
watch classic TV shows in glorious 4K resolution.

~~~
drcode
What I want is classic movies in full 360 vr... I'm guessing this will exist
in a few years, at the current pace of things.

------
Keyframe
What is the state-of-the-art in segmentation mapping in video?

