Video2NAND – Abusing video codecs for great computational power

fch42 3 hours ago |

Mapping the abilities of (fixed-function) "stream transformers" to compute is old, much older than "GPU libraries" or render pipelines and all that. I remember the Commodore Amiga used the blitter's combine to decode/encode floppy data, but there are likely other examples predating that. In which sense is VP8 "different" there from another "fixed-function" processor?

pseudosavant 14 hours ago |

Cool that you can make a NAND gate like this, but I was more hoping for something that showed how to practically (ab)use a video encoder for "great computational power" to do something real. Not just how to make a NAND gate with one.

tverbeure 14 hours ago | |

I was hoping for some variant of the FORCEDENTRY iPhone Messages exploit, which built an entire CPU with the PDF fax blitting operations.

pseudosavant 13 hours ago | | |

Fair enough. Maybe not that far, but perhaps do a little one thing with these NAND gates they theorized? My issue may even be more with the HN title than the blog post, because the post never suggests anything like using a video codec for "great computational power".

thenthenthen 12 hours ago | | |

I tried looking it up but failed to find any technical write ups of how this was done, do you have any links maybe? Super interesting to read.

bawolff 12 hours ago | | |

https://projectzero.google/2021/12/a-deep-dive-into-nso-zero...

zahlman 6 hours ago | | |

I wonder how much overhead that incurs (i.e. how much slower the exploit code is in the JBIG "VM" versus native).

nomel 12 hours ago |

Looks like VP8 has a max resolution of 16383x16383 pixels. Looks like minimum block size is 4x4 pixels, so 4096x4096 blocks, with maybe 7x7 blocks per gate, including some wiring, so maybe something like 500x500 logic gates?

I don't think we'll be running Doom, but you could definitely make a little CPU, with programs!

picture 10 minutes ago | |

What is this math from? How do you suppose there to be less gates than pixels? Also, different implementations may unroll to different degrees, with some lower thruput hardware doing stuff sequentially by reusing the same logic

akoboldfrying 3 hours ago |

I was wondering whether there's any way to make wires "cross", which would be needed to construct a non-planar circuit. But even if there isn't a way to do this directly, Gemini tells me that it's possible to construct a planar circuit (built out of XORs which are themselves built out of NANDs) that effectively "crosses" two wires!

moralestapia 9 hours ago |

Funny this is how people initially started re-shaping some computational problems to resemble graphic pipelines and get "great computational power"; back when it was called GP(General Purpose-)GPU computing.

arjvik 13 hours ago |

Sadly doesn't look turing complete (yet) :(

mrheosuper 12 hours ago | |

if you can make a nand gate, you can make any other logic gate, thus turing complete

phire 10 hours ago | | |

I think you also need some form of intermediate storage to be Turing complete.

Not that this example doesn't have intermediate storage... But I suspect you could create or discover a mechanism that is expressive enough to implement any chain of logic gates, but is incapable of expressing storage (can't even use the gates to implement a latch)

I'm also a little dubious of granting the "Truing complete" label to something that can't conditionally terminate (at minimum)

Maybe you can trick a video codec into looping forever without new key frames (in which case it can never terminate), but most likely this would need to be implemented unrolled, so always terminates after a fixed number of iterations.

akoboldfrying 3 hours ago | | |

It looks like you can only make "wires" that go down or to the right, so there's no way to connect up NANDs to make a latch or flip-flop (which would require wires going back up or to the left).

phire 3 hours ago | | |

If you were limited to just a single frame of itra blocks, then yes. This would be an example of logic without storage.

Also quite limiting as wires could never cross.

But I was kinda jumping ahead and assuming VP8 allows you to mix and match intra and inter prediction modes within a single frame (and that inter prediction will feed into intra prediction... that might be a flawed assumption...)

Inter prediction should allow you to copy from any block on the previous frame, effetely creating unlimited length wires in any direction. This also makes the previous frame latched storage, without having to construct a latch from gates.

akoboldfrying 12 minutes ago | | |

> Inter prediction should allow you to copy from any block on the previous frame, effetely creating unlimited length wires in any direction.

I think "going inter" just gives you a third "dimension" along which you can still only travel one way -- but you need bidirectional travel (outputs feeding back into inputs) to implement memory. It makes sense to me that bidirectional travel is not possible here, since it would necessitate some kind of "keep processing until convergence" that could (and often would) prevent the decoder from making progress.

> This also makes the previous frame latched storage, without having to construct a latch from gates.

This makes me think we have different ideas of what "latched storage" means. I think the block that you would call a "latch in the previous frame" is functionally no different from a block elsewhere on the current frame? I don't see how it could have the same "address" but store a different value over time, which is what I'd call the defining property of all "storage".

firer 2 hours ago | | |

Exactly! Hopefully going to get around to it next weekend :)

Also, although you can't trivially cross wires, you can create a wire crossing using a few XOR gates [1]

[1] https://cs.stanford.edu/people/eroberts/courses/soco/project...