Also, we need both mechanical keyboard to sense key presses and 3D tracking of finger movements? I get it, Logitech wants to keep selling keyboards, but for VR experience I would rather have tracking of facial features and eye movement.
[1] https://d201n44z4ifond.cloudfront.net/wp-content/uploads/sit...
Perhaps some combination of voice, gesture, and 3D could be useful specifically while in-game. But for productivity, I don't think there's anything better than a traditional workstation.
Voice has privacy issues. Gesture recognition is inaccurate and causes "gorilla arm" syndrome.
3D doesn't really add much, and in fact for productivity some people think even 2D Window-based GUIs are cumbersome, and that is the entire motivation for tiling window managers. From that perspective, 3D is even more chaotic and cumbersome.
QWERTY
All that needs to be said about the difficulty of changing a primary interface method.
Have you ever watched someone who is handicapped who has eye tracking for a keyboard? Have you paid attention to how slow and error prone it is? What on earth makes you want to have THAT as your primary input method???
I thought about input interface that could replace keyboard for VR and mobile devices. Brain implant would be perfect, but that's not feasible with today's technology. Voice is good contender, except for privacy issues, and it's not really usable in work environments. Keyboards are our best solution so far, but they don't evolve and to me they seem like a dead end (plus, they are not portable and require both hands to use effectively).
I was thinking about touch sensitive surface that recognizes drawn glyphs. The idea is that anyone who is literate can start using it without any training, just draw letters instead of typing them. With machine learning and some clever visual feedback, both user and machine could adapt to each other to increase the input speed (by simplifying glyphs and by defining 'snippets'). Interface could be expanded to both hands to double the speed, but it would be completely functional with just one hand. It could even be used blindly. For mobile devices, such sensitive surface could be placed on backside - virtual keyboards that take half of screen are just horrible.
One of the first things I did with my Rift (well, after Superhot...) was try out the Virtual Desktop kit.
It's lacking. I thought it'd give me a 3D frustrum to throw window back/forward, drop Spotify out to the peripheral, have VisStudio in glorious megapixel size.
Instead, you get your regular desktop screens mirrored and constrained to a 2D plane. Gods help you if you've got mismatched DPI screens, like my 1080p & 2160p pair.
(Go on, I'll await some smart sod to tell me "Product" is exactly what I'm looking for...)
Using new technology to incrementally improve existing ideas and techniques usually works better than trying to start completely from scratch.
VR should be considered as completely new medium. 2D windows in VR should only be used as backwards-compatibility layer. Re-inventing every part of interface at the same time sounds great to me. Let's make as many competing solutions and then pick ones that work the best. This paves the way not only for rich VR experience, but also for AR.
I spent a few hours the other day searching for existing chording keyboards and they were all really clunky.
...plus the QWERTY keyboard design goes back to 1870 ( https://en.wikipedia.org/wiki/QWERTY )
While just putting a window on a nebula background works as a Proof of Concept, it does need further innovation. The keyboard, however, does not.
The problem with the gif is that it doesn't make any use of the "virtual" environment, but this is where vr should be heading at least as the intermediate before fully interactive vr.
That, and porn
No. You don't get it. This doesn't do 3d finger tracking. It just figures out where the keyboard is and does some edge detection to give a rough idea of where your hands are.
The other day I was playing a game of onward. It was the rare instance that a team wanted to work together on a shared goal. In the 15 seconds before the match, while planning some tactics... it occured to me how natural the meeting felt. It was like we were all in the same room. I felt engaged with EVERYONE. That doesn't happen on conference calls.
I work from home full time, I think the biggest downside are the missing meaningful interactions with other people. I think VR has huge potential to bridge that gap.
Personally I think we need to reimagine interfaces to the world around us, even the virtual worlds (VR/AR/MR) around us. Voice input, AI, hand sensing technologies could make for new ways for changing the worlds. The book Daemon by Danial Suarez and his eSpace holds much promise. I have been experimenting with those idea in a new VR world I have been working on for a few years.
And actually...that seems like a portable idea. So I suppose you could do this for any tracked thing.
EDIT: You can simply use the Vive's camera, with tracking.js color tracking, especially with a small minDimension (number of pixels) threshold. Yellow is good.
https://artoolkit.github.io/jsartoolkit5/examples/ https://trackingjs.com/examples/color_camera.html ; http://www.keyboard-layout-editor.com/#/ https://joric.org/keycaps/#GB-Retro-DSA etc
but i'd would rather use a wider range of motion (at some point, if someone makes that) to input to stay active.
Old keyboards were (generally) designed to allow for touch-typing. Modern keyboards are (generally) designed to be looked at.
Btw, this is a VR keyboard from the Dennou Coil anime 2009. Nobody was using real reality (pun intended) laptops there, only VR ones. I guess they were running on some AWS like cloud.
https://giphy.com/gifs/typing-keyboard-dennou-coil-1337mjZhd...
VR also has the potential to be way more addictive than current forms of online interaction and entertainment.
I'm reminded of I think it was a Larry Niven story (maybe Ringworld) in which the protagonist has an electrical wire put in to the pleasure center of his brain and he just sits home and pushes the button that activates it for weeks on end.
Technology is getting closer and closer to that.
The book briefly explores the implications of this, including how the hierarchy of human needs is met in a VR-centric society.
He only disconects the droud because he knows he will die from starvation, dehydration and atrophy if the doesn't go through the motions daily. But every moment without the droud is existential dread.
Considering humans can ride bikes, drive cars, play instruments, etc. (including typing on keyboards!), I think that indicates that non-verbal, physical interaction is not nearly saturated as a transmission channel.
Conversely, it's hard to imagine someone verbalizing "navigate to HN" in a loud open-space office, or "Excel, create a pivot table" or whatever. I think it's fine in private spaces like your home, but in public spaces, you're implicitly broadcasting your activity to everyone around you, which I consider to be a strong negative.
Is subvocalization a possibility? Mic or EEG setups might need to be slightly different.
I don’t personally believe in voice input. English is my second language but I have spoken it daily for close to 10 years. I can’t get Siri to understand what I say so I just keep it disabled and iPhone keeps bugging me to enable it.
We cannot rely on voice input as that is a dead end. Until we figure out how to link thoughts directly to computer inputs keyboards will be the king.
If you want to sculpt a model in the real world, you'd use clay and your hands. We can use VR to move away from KBM interaction for things like that.
If you want to write a book in the real world you...type it out. Its the best way we know to get text out of your brain and somewhere else. Don't throw the baby out with the bathwater.
I used to hope for the unlimited monitor space, now I am not so sure how practical that really is.
I could invest a lot of time to unlearn my current way of typing and learn proper touch-typing technique, but that's a lot of work and doing it just to be able to type in VR feels like a waste of time. With this tech I'd be able to work in VR without changing my way of typing. To me, this makes using VR for work practical for the first time. It's actually huge, and if it works well it's, to me and others in similar situation, potentially life-changing tech.
It's hard to place your hands on the keyboard in pitch black or when you're in a weird position like lying on your back with the laptop on your chest. If you've ever partially closed your laptop so that the monitor would shine on your keyboard so you could find 'f' and 'j', you know exactly what I'm talking about.
And also TFA says "for a true typing experience you need to see your hands, and we’ve created a way to use the Vive’s existing tracking to do that".
As long as I can use such markers to position my hands in the proper initial position, I really don't like to have a lit keyboard in the dark. It's distracting and annoying.
If your keyboard doesn't have such bumps it's easy enough to make them either using some craft materials you can stick on the keyboard (like some tape even) or by cutting small notches in to it using a file or a dremel tool.
“But VR can transform and augment that trusty keyboard – so easy to disregard – into a contextually aware companion for whatever application you use, becoming a palette for your creative workflow, dynamically providing you with any commands and shortcuts you need.”
But that's on a physical keyboard that has something which virtual keyboards lack: tactile feedback.
Tactile feedback is one of the missing pieces of today's VR devices. Without it typing on a fully virtual keyboard is just not going to be nearly as accurate as on a physical keyboard, and providing visual feedback is just about the only thing you can do about it.
If you are playing a VR game where you are using a joystick or some other peripheral, you still may need to use your keyboard for some actions, in-game chat, or simply to pull up a browser and google something while in-game.
If you primarily have your hands on a joystick or game controller, you would have to take your VR headset off just to figure out where your keyboard/mouse are.
the keyboard can follow the user's view... along with the silhouette of the user's hand...
That said, I don't really have a better suggestion. Typing in VR is just going to be plain painful for anyone who can touch type with any speed on a physical keyboard.
Citation needed. (Seriously, I was looking for statistics on that recently and couldn't find any)
Although, the costs of switching outweigh the benefits for almost everyone, except for those with repetitive stress injury.
I think you're throwing the baby out with the bathwater here. There are plenty of teams working on brand new UI/UX experiences in VR, and I'm sure that in time the old 2d paradigms will have more competition. But also, I still choose to use a terminal for git when I could use a GUI, so I'm not actually convinced that "2D windows in VR" are going to or should go away.
The OP shows a great step towards being able to replace your workstation with some VR goggles, a keyboard, and a CPU. That prospect excites me more than any "new medium" VR experiences that I can see on the immediate horizon; being able to work from a beach hammock and not compromise on workspace efficiency is incredibly appealing.
I'd like a touch sensitive surface... on a keyboard.
Consider an expert keyboard user. With a nice keyboard. Adding downward-facing hand tracking, such as a head-mounted Leap Motion, adds value. Now you can gesture on the keyboard, and above it in 3D. Permitting a very rich input vocabulary. And, for example, using the entire keyboard as a touch surface. Except... while fingers are highly sensitive to tactile contact, neither traditional keyboards, nor current hand tracking, can provide good contact information. Finger-tip position tracking isn't quite good enough to infer contact existence and pressure. So while you can use say the J-key keycap surface as a trackpad, and even infer touch-untouch events from gross finger motion (better than requiring a keypress, but not by much), you still can't get light-touch events (eg "I was clearly pressing harder when I stroked down, but just skimming when I moved back up - I clearly felt the difference, so why didn't the keyboard?") So, I'd love a contact sensing mesh which could be laid over an existing nice keyboard, without compromising key feel. I can get position information elsewhere, and keypresses from the keyboard of course, but there's no existing source of multitouch contact, let alone pressure. Any ideas?
