I'm excited to show case an update to a personal project of mine. Its called ffmpeg-over-ip and it allows you connect to remote ffmpeg servers, so if you have one machine with a GPU (could be your windows gaming laptop, gaming PC, a macbook?) and a machine (or VM, docker container etc) without a GPU, you could use the remote GPU to do GPU-accelerated video conversion.
The way it works is pretty neat, there are two components, a server and a client.
- The server (has the GPU) comes with a patched up ffmpeg and listens on a specified port - The client (without the GPU) connects to the server, takes the file IO requests from the server and runs them locally.
ffmpeg doesn't know that its not dealing with a local filesystem, so this approach works with multiple inputs or outputs like HLS, and is perfect for home media servers like Plex or Jellyfin or Emby.
One server can offer up its GPU to as many clients as needed, all clients offer up their own filesystems for their requests, the server also comes with a static build of ffmpeg bundled (built from jellyfin-ffmpeg scripts) so all you have to do is create a config file, set a password and you're good to go!
It's been about a year and half since this was last submitted (https://news.ycombinator.com/item?id=41743780). The feedback at the time was around the difficulty of sharing a filesystem between the machines so that should no longer be a problem.
This has been really useful in my local setup, I hope you find it useful. If you have any further questions, the website has some FAQs (linked in github repo), or you could post them here and I'll answer them for you!
Thanks!
The project is https://ffmpeg-commander.com for generating ffmpeg commands, but with an experimental backend to offload the tasks.
Do you support chunked encoding across multiple servers? It would be a great feature to support larger video files.
Was really impressed by your work on Pundle. (It was an amazingly fast HMR dev environment - much like Vite today.) Felt like I was the only one using it, but it was hard to walk away from instant updates.
It's even reduced load considerably because most of the time the disk doesn't even need to be touched.
ffmpeg-server runs a patched version of ffmpeg locally, ffmpeg requests to read some chunks (ie give me video.mp4) through our patched filesystem (https://github.com/steelbrain/ffmpeg-over-ip/blob/main/fio/f...), which gets sent over the same socket that the client established, client receives the request, does the file operations locally and sends the results back over the socket to the server, and server then sends them to ffmpeg.
ffmpeg has no idea its not interacting with a local file system
- https://github.com/steelbrain/ffmpeg-over-ip/blob/main/fio/f... - https://github.com/steelbrain/ffmpeg-over-ip/tree/main/patch...
I am not sure they'll be accepted for upstreaming, but in exploring the options, I noticed ffmpeg has sftp:// transport support and there were some bugs surrounding that. I do intend to publish some patches for those.
A single CPU core on a 9500T or a Ryzen V1500B is fast enough to real-time re-encode 60mbps 4K H264 to 1080p 5mbps h264, aka, for a core use case - transcoding for web for Jellyfin over cellular, for example - you haven't needed hardware video engines on PCs for 9 YEARS.
I have no idea why people are so hung up on hardware video encoding. It's completely wrong. The quality is worse. The efficiency is a red herring - you will still use every CPU core for IO threads in ffmpeg, if you don't configure that away, which you do not. And it requires really annoying setup and premium features on stuff like Plex. It just makes no sense!
If latency is important to you, well then hardware engines make sense. But you are throwing away the latency sending it over the network. The only use case (basically) is video game streaming, and in that case, you'll have a local GPU.
I have never read one of these ffmpeg network hardware encode innovations to have an actual benchmark comparison to single thread software transcoding tasks.
I know you mean well but really. It makes NO sense.
I would love to learn more about this! What can I do to fully optimize ffmpeg hardware encoding?
My use case is transcoding a massive media library to AV1 for the space gains. I am aware this comes with a slight drop in quality (which I would also be keen to learn about how to minimize), but so far, in my testing, GPU encoding has been the fastest/most efficient, especially with Nvidia cards.
People who need this know who they are. Not everything is for everybody.
Dammit I really wish Plan9 had taken off. It isn’t perfect but it does a much, much better job of helping me run my applications in ways that I want.
If anyone doesn’t already know, one method of Plan9 remote access is to “cpu” into a remote machine which has the hardware you need. Your local filesystems go with you, and your environment on the remote machine consists of your local filesystems mounted to the remote machine, but only for you, and all applications you run in this context execute on the cpu of the remote machine and have access to the remote machines hardware, but your local filesystems. Imagine SSHing into a remote machine and your entire environment goes with you, development tools and all. That’s what Plan9 does for you.
So if I “cpu” into a machine without ffmpeg, but with a GPU and I run ffmpeg, not only will it work, but I can tell ffmpeg to use a hardware encoder with a simple command line flag, and it’ll work.
The cpu command does it all gracefully, and it’s already written, it even works on Linux.
Had you thought about using FUSE on the server side, rather than patching ffmpeg? Like a reverse sshfs? That avoids patching the ffmpeg binary, which allows usage of wierd and wonderful ffmpeg binaries with other people's patches.
I'd be interested in seeing how well it works with SBC GPUs - many have hardware decoding and encoding, and their vendors love to fork ffmpeg.
That was the one motivation, the other one was that it would require rewriting arguments going into the server. What you're describing was essentially what ffmpeg-over-ip v4 (and its earlier versions!) was, and the constant feedback I'd heard was that sharing filesystems is too much work, ssh servers on windows and macOS are a bad experience, people want to use a bundled solution.
Forking ffmpeg was no easy task! Took forever to figure out the build process, eventually caved in and started using jellyfin build scripts, but that has the downside of being a few versions behind of upstream HEAD.
I was thinking of the server end of an ffmpeg-over-ip system bringing up a FUSE filesystem backed by something similar to your VFS-served-by-the-client. Combine that either with argument rewriting, or chrooting into the FUSE filesystem.
As another commenter said, where's plan 9 when you need it? If you go the FUSE route there are existing 9P implementations for both server and FUSE client you can use.
You can mix and match operating systems, macOS, Windows, Linux, you do not need sudo privileges.
rffmpeg needs a shared file system which could be a huge pain to setup: https://github.com/joshuaboniface/rffmpeg/blob/master/docs/S...
ffmpeg-over-ip patches ffmpeg and only needs one port open for the server, then you just run the binary, no mounts needed at all.
FFmpeg is mountains of extremely complex C code whose entire job is processing untrusted inputs.
Choosing to make such code network-enabled if you can't trust your inputs, I would recommend to sandbox if at all possible. Otherwise you are asking for trouble.
The usecase for something like this is when you control both sides, server & client. There is some basic HMAC auth built into each request.
> I would recommend to sandbox if at all possible.
Since the server is a standard binary that doesn't need any special permissions, you could create the most locked down user in your server that only has access to a limit set of files and the GPUs and it'll work just fine. This is encouraged.
I've been reading hacker news for ages, but somehow I missed that there was some convention relating to showing technology on hacker news.
I would have very much loved to use the built-in capabilities instead of patching ffmpeg to add a vfs layer and spend a ton of time figuring out the build pipeline once you add all the codecs and hwaccels. I do hope to be able to change this in the future, I've identified several bugs that I intend to submit patches for.
For linux, I am thinking of adding convenience helpers around systemd service installation
[1] https://docs.joinpeertube.org/maintain/tools#peertube-runner
- Client makes request to server (which opens a bidirectional network socket)
- Server uses that bidirectional socket, spawns a local patched ffmpeg with vfs-like characteristics
- ffmpeg (using client-server bidrection socket) does input/output operations, treating client filesystem as if it was local
Thus client doesn't need to open any ports, or expose its filesystem in a traditional mounting manner, and one server can handle filesystems & requests of any amount of clients.
If you want to optimise, try different encoders (sounds like you've already done some of this) and lots of different settings - it'll involve a lot of tuning if you want to figure out the right balance for your particular media between quality/speed/size, while also making sure that your machine hurts as much as possible.
Driveby 2c as a video industry person: don't retranscode your media unless you've got them in a really space inefficient codec and you're seriously hurting for space. You'll burn a lot of power retranscoding, are you actually saving useful $$$ of storage in exchange for that spend? Storage is cheap, and there's always a better codec coming along you could retranscode into and save some more space. It's a vicious cycle: each generation has to encode the artifacts from the previous generations.
Maybe my use cases for ffmpeg are quite narrow, but I always get a speedup from moving the files off my external hard-drive, suggesting that is my current bottleneck.
The hope is that you stream over LAN not the interwebs!
> I always get a speedup from moving the files off my external hard-drive
Based on your description, it does seem like your ffmpeg may be IO limited
I'd argue this is for nobody haha
Nobody using jellyfin plex or whatever needs it: they should just use software transcoding, it's better in pretty much every way.
- I dont want to unplug the GPU from my gaming PC and plug it into my linux server
- Then: I dont want to figure out PCI forwarding, I'll just open a port and nfs to the containers/vms (ffmpeg-over-ip v4 needed shared filesystem)
- Now: I have a homelab of 4 mini PCs and one of them has an RTX 3090 over Oculink. I need it for local LLMs but also video encoding and I dont want to do both on the same machine.
But you've asked a more fundamental question, why would people need hardware accelerated video decoding in the first place? I need it because my TV doesn't support all the codecs and I still want to watch my movies at 4K without stuttering.
My usecase is just-in-time media transcoding, I'll see if PeerTube remote runners support it
Sometimes you want faster-than-realtime encoding, such as when backing up your video archive.