State of multi-player Wayland(blinry.org) |
State of multi-player Wayland(blinry.org) |
You can find installation instructions in the script of last year's workshop: https://blinry.org/cursor-workshop/
The way screen sharing solutions like Teams handle the multi cursor thing is to just keep the main cursor jumping around, which works, but meh.
Having multiple cursors within the same environment theoretically enables such interesting collaborative experiences. Imagine a large surface in which everyone has their own workspace (cluster of windows), but you can seamlessly go between them …
See, when you start a graphical session on modern systemd-based Linux, a new logind session is created, and it is immutably tied to a seat. So you can't just remove the seat from a running logind session and then make it remote, or add the seat to a running logind session and make it physical. I actually tried patching systemd-logind to do this, but when I proposed it upstream, Lennart Poettering himself actually took the time to explain that this wasn't the intended way to do it: instead, you would actually just create a new logind session and drop the old one entirely, or literally whatever you want.
Which to me sounded perposterous, but then I realized that it really wasn't that ridiculous. As it was, when you boot a desktop using systemd-logind, the desktop itself doesn't typically run much of anything directly in the session context anyway - the compositor and almost all services actually run as user services, as in systemd-land having more than one active desktop session per user isn't really supported. Okay so we can do this!
And so I set out to prototype this with KDE, but there are still a few issues...
- First and foremost, $XDG_SESSION_ID. Believe it gets set by the systemd PAM module somehow? I don't fully recall. Either way, it gets used by various programs to determine the current logind session, but if our graphical session can "roam" between them we need a mechanism that can handle changing over time. I propose we stuff the current logind session ID into a file in $XDG_RUNTIME_DIR instead.
- Secondly, compositors need to be redesigned to handle this. Compositors today can hotplug displays and input devices - that's a good start. Unfortunately, at least kwin can't switch backends at runtime. You can really go one of two ways, and I tried a bit of both and either can work but both are a bit ugly: one is to make a single backend that can handle either case and the other is to allow switching the backend at runtime. Either way, when running remotely the card* DRM nodes are unavailable so you need to switch to renderD* nodes instead. You can also use both, and only use card* for scanout when available, but it seems a bit trickier.
- Finally, I genuinely have no idea how this could or should be handled in non-systemd environments. I have admittedly forgotten how that all works. It would be rather lame to simply require logind for this to work, but sadly it is the path of least resistance at least at first.
And maybe some more, but otherwise the concept does indeed broadly work. I mainly have only tested it in virtual machines with DRM native context, but I can see everything working switching in either direction at least once before things start to get screwed up. Even OpenGL clients and audio playback can move seamlessly! It's just like what Windows accomplished over 20 years ago!
Aside from convenience and being cool, it would also solve a couple problems. No need to worry about the limit of only one session per user: just move your session with you. No need to worry about your physical session being unlocked when remotely connected: when you log in this way your physical seat gets disconnected from the session entirely and the display manager takes it back over.
I actually would really like to at least make some progress on ground work that would be needed to make this happen, although I'm not really that sure that anybody cares. But either way this post's brief discussion about physical vs logical seats activated a few neurons so I couldn't help but dump this out.
It's not obvious to me. This looks like software ageism. :)
There are at least 2 things that could be done to improve performance in the author's setup:
1. Use the --gpu flag which enables dmabuf capture and h264 encoding via vaapi.
2. Connect using a dedicated VNC client like TigerVNC. This reduces latency by roughly 50 ms.
That being said, noVNC has gained hardware decoding of h264 via WebCodecs since the writing of that blog post, so performance will have improved some. It's unlikely that the 50 ms latency has gone down, but it might be worth measuring again since browsers may opt to render video through a dedicated overlay plane that bypasses regular rendering. However, it would be silly not to synchronise frame presentation between overlays.
1. Hilarious security. Passwords silently truncated to 8 characters, etc.
2. Inefficient graphics. Very laggy on all but the best connections. Just sending an H.264 video stream is better these days.
3. No seamless window mode.
4. I've never needed it tbh but I doubt it supports remote sharing of USB, printing, audio, GPU etc
1. Wrong. There are many extensions that improve upon authentication and encryption.
2. This mostly depends on the implementation, and not necessarily the protocol and/or its extensions.
3. This would require deep integration with the window system on both sides, so I doubt that this is even a thing with RFB on Linux. There certainly aren't any wayland protocols to do this.
4. No one has ever asked me to implement any of that with the exception of audio, but there is also an extension for that.
https://metavnc.sourceforge.net/
TLDR: it leveraged a specific client that could
a) connect to multiple vnc instances b) display them in a single window c) act as the "window manager" / "desktop environment" - doing this by enumerating menu entries from each instance and creating a composed whole. Enumerating the active applications from each instance and composing them into a single "task bar"/"window switcher" interface.
this is probably close to 20 years old now.
And you might be right about that ageism comment… didn't realize more efficient encodings were a thing now! Thanks for your work! :)
You might enjoy the KeyMouse (https://www.keymouse.com/).
… unless you do want the left half of your keyboard to type into one window, and the right half of your keyboard into another.
Even though there is only one cursor it's been a really powerful way to work, being able to control the mouse with either thumb.
Then there are the alternative keybindings that allow moving the mouse using the wasd keys as well.
> I think there could be a term that’s describing this concept even better – in the title of this post, I picked “multi-player”, to make clear that it’s mostly about multiple people who are collaborating. I’ve also seen “multi-cursor” or “multi-pointer” (which are missing the keyboard component).
I'm not aware of actual hardware that would make this feasible, but if you invent a touchscreen which recognises between two people's different fingers, you could have a client which tracks who clicks where without any protocol changes.
Come to think of it, Steamdeck is Wayland? How do they implement it?
Why? That seems like it would make it a lot harder to move the cursor precisely. (in fact the main implementation of it on linux I know of was a joke in IOCC)
I get the impression that you're thinking of a much jankier implementation of "inertia" than that, and to be honest I wouldn't be surprised if that's what we'd get anyway.
An unambitious inertia describes the linux community of the past decade pretty well.
In firmware.
> X11 (the X Window System) has its own multi-seat extension called Multi-Pointer X (MPX), which I also had a lot of fun with a couple of years ago.
Enabling in X works via:
xinput list # find second mouse id
xinput create-master second
xinput reattach <mouse-id> "second pointer"
xinput reattach <mouse-id> 'Virtual core pointer'
xinput remove-master 'second pointer'
In fairness to the creator, it's a niche product that would have taken quite some time to develop.
I quite like BenVallack's video about the Svalboard [1] (and his videos in general about his journey through the world of split keyboards).
I hadn't heard of a mouse either but when the trackball appeared, mainly to play Centipedes and Missile Command, I used it as a single-button mouse anyway, not only to point & click but to drag & drop which I had never seen demonstrated.
So I made a little 2-player game, not fancy, each player stayed in their own window.
Of course nobody had ever heard of "Windows" either since MS-DOS wasn't even out yet.
The typical GTK application out there can't handle multi-touch and can rarely even handle several mouse events (e.g.: mouse-wheel scroll while dragging), so I doubt they have full blow support for multiple seats.
I thought this is a bug in the Wayland implementation: I find it utterly unintuitive to work like this. (I just have no idea with whom to file a bug report, which is why I haven't.)
Are you implying that this is intentional behavior?
This is about actually having multiple, independent mouse cursors on the screen at once, with different input devices controlling different cursors.
I have assigned my touchscreen to be in a separate seat. This allows me to interact with apps without moving my cursor or changing my window focus.
For instance: web browser on the left, with a document. Text editor on the right. I am writing some notes with my keyboard, scrolling the document with my touchscreen. The editor stays focused, no back-and-forth with the mouse. It feels very natural this way, instead of having your cursor jump where you touched the screen and change your M/K focus.
Like: how would one support multiple keyboards? Is there an experimental or logical model for that?
Multiple mice is esoteric.
It's an incredible hacky piece of software that does work by putting together like 10 oss. Incredible and cool.
They have some UX around this
The one other constraint that I'd really like to see relaxed is about displays, about being able to merge and subdivide screens as I please. Having a huge ultra-wide display really would be so much better if I could divide it up somehow. This is kind of the headline feature of stilch, which is a pretty neat compositor, https://github.com/wegel/stilch .
Niri has some tickets on it too. Merging monitors into a big display, and splitting a display (moments before the multi-pointer ticket was filed in Niri), https://github.com/niri-wm/niri/discussions/1285 https://github.com/niri-wm/niri/discussions/3160
There's been all kinds of rough incredibly hard things going on with Niri just trying to figure out how to handle some of the more interesting copy-paste scenarios, and with really weird sequencing issues across basically ephemeral inputs, that have really plagued the project and made life incredibly bad for downstream apps. This is just my opinion, man, but I think the lack of ambition about these ridiculous over the top features eventually starts hampering what turns out to be relatively prosaic asks. Better models for multi-seat, multi-cursor help scenarios like remote-desktop, and video-chat remote control, that at first don't seem needed, but are fundamentals that pay dividends if you shoot for them.
Not sure the best level to implement this. There’s probably already solutions for this now that I think about it…
> There certainly aren't any wayland protocols to do [seamless windows]
I think wprs can do this (it calls it "rootless") but I haven't actually got around to trying it yet.
And at home, I have a mouse on the left side of the keyboard and another one on the right.
Assuming that multiple mice obviously means multiple cursors while in the same comment assuming that multiple keyboards obviously does not imply multiple input cursors is in fact pretty confusing to me
So a keyboard being attached to a seat, whose input might be associated w one active window, while another kb is associated with a different seat and different active window. Perhaps I'm just overthinking it.
Interestingly enough, it seemed that the inputs from multiple mice onto the same cursor is summed up by default. So you could fight cursor movement you disagreed with by moving the mouse in the other direction.
Very silly, but quite funny.
The fact you're griping about implementation details like pipewire makes me think you also tried micromanaging your env and broke something. If you use a fresh install and leave everything default, I'm sure it would work for you like it does for everyone else. I use KDE too btw.
All the "micromanaging" you need to do to not have screensharing is one of the following; 1. Don't have a started dbus session 2. Don't have portals installed and setup 3. Don't have pipewire installed and setup
Meanwhile on X what you need is: 1. Have X
So no, it wouldn't work, the reason why it works for those people in the Reddit post is because they use distros and sessions that do all the things I listed that make screensharing not work if any one is not done.
PS: Wayland does have testing protocols for screenshots and screenrecording like 'wlr-screencopy' and some DMA buff one I forgot the name of. But for screen sharing there's nothing that Wayland implements itself afaik.
Or, did you remove some core OS services, and now you're complaining when various things started to break?
PERSONAL Computer