realvr
patreon
The state of VR
🕑 Added 2022-04-20 07:55:14 +0000 UTCOr, "You can put your hands down now, citizen" []-D
I was recently interviewed by Tomislav Bezmalinovic from German website https://mixed.de/ about my Patreon, VR modding and the state of VR gaming in general.
Tomislav's questions led to a rather interesting discussion, which of course had to be summarized in the published article, so I've decided to put the full transcript to the original interview here. I'm also opening this post up for public discussion, as I believe the points we touched are of relevance to the VR industry as a whole, not just to my Patreon mods.
Hi, I'm Tomislav and I've been covering the VR and AR industry since 2016 for the leading German XR outlet MIXED, which as of recently also has an English version at https://mixed-news.com/en/. The articles are automatically translated with machine learning.
I would like to write a short portrait of you and have a few questions.
I saw that you have almost 2,000 Patreon supporters. That's quite a number and shows how important VR modding has become. So, what's your background?
Hi Tomislav, thanks for reaching out! I'm grateful for your interest, and quite happy to provide answers to all your questions.
Before plunging into this modding adventure I was a freelance developer and software engineer. I've also been a gamer all my life and I always loved new technologies, especially when they allowed the gaming experience to become more immersive, like with 3D glasses. So VR was just a natural fit and I became passionate about it right away. I was one of the original Oculus Kickstarter backers.
How come you started developing VR mods?
It happened almost randomly. One of my favorite games a decade ago was No One Lives Forever 2, a now almost-forgotten gem by master designer Craig Hubbard. The game code was open-sourced at some point in the past, and I had already worked on it when NVIDIA 3D Vision was all the rage, in order to make the game compatible with stereoscopic 3D rendering. Thus, when VR became available, I decided to see if I could mod the game to transform it into a native VR experience. That was back in 2017: my old thread on the Oculus forums is still visible here and the mod can be seen featured in a video by SweViver.
The reaction from VR users was very friendly and encouraging (I even got to chat about the conversion with Craig Hubbard himself!) and that prompted me to later tackle a much bigger and complex game: GTA V. My free GTA V VR conversion has been downloaded by more than 150K users! After that I just decided to take the plunge and see whether I could become a full-time modder.
Is there a team helping you or is it just you?
It's just me for the moment, although things are growing fast! There's recently been talks about creating a Chinese community, and I don't even speak the language, so I will probably need some help pretty soon :-) My mods also attract a lot of attention in Germany, and it would be great if a local community could be started there too.
Why do you think the big companies don't port their big games by themselves?
It's a sad combination of many factors.
VR as a technology has been around forever: most people don't realize it, but the first HMDs came out in the 60's. However, the current wave of interest became popular between 2012 and 2016 when Oculus came out with their first consumer models. In those magic years, everybody started experimenting with VR. Studios were born around ideas, publishers were thinking about investing, there was sort of a gold rush.
But it all happened too suddenly, and soon several blocking factors came out:
a. adoption was slower than expected, so returns on investment was very poor or nonexistent for many studios;
b. hardware and software requirements, as mandated by some "purists" who were too enamored with the tech and not sufficiently concerned with the reality of the market, were way too steep. 90 Hz (in stereo, so actually 180 fps!) at the high resolutions and FOV needed for VR were out of the realm of possibility for most game engines on one side, and for most accessible video cards on the other;
c. the introduction of motion controllers (virtual hands in game) was ill-timed and put the final nail in the coffin. I know that's a wildly unpopular opinion, but that doesn't make it any less true.
Game developers were suddenly expected to create games using a completely different paradigm than they had perfected for many years in the past: everything in their games was now supposed to be interactive and respond to the presence of your virtual hands; shooting and reloading mechanics were expected to be utterly redone for VR; driving cars needed to happen with imaginary hands on an imaginary wheel; even inventory management supposedly had to be redone by making use of invisible pockets and backpacks and holsters.
So, studios and publishers reacted in a very natural way: when pressured to deliver a completely different product, tailor-made for a niche market that was still in its infancy and could guarantee no big returns, they simply clammed up and decided to ignore VR as a passing fad.
With that many Patreon supporters you must have a good income. Are you having another job beside VR modding and if you do, how do you manage that?
When I started the Patreon, my dream and stated intention was to put my freelancing work on hold and devote all my time to modding. Fortunately, as you said, the community reacted positively enough, and at least for the moment I can afford to do just that. It's not guaranteed to last forever of course, and there are many factors that could still turn this adventure into a commercial success or failure. But as long as people are interested in playing huge open-world games in VR, I will keep doing this!!
Which VR project or achievement are you most proud of?
It's hard to tell. Every project feels a bit like a child to me, and I love them all equally :-) But the two VR conversions that enjoyed the most widespread success for now are without doubts Grand Theft Auto V and Cyberpunk 2077.
Would you like to get hired by a VR studio or company at some point or do you want to continue working on your own projects?
To be honest I'm not looking to get hired. I had a few interesting offers, and I will certainly consider any proposals that should come my way with the attention they deserve, but I feel that I can have a much larger impact (and with greater professional satisfaction) as an independent modder.
After finishing your current projects, what's coming up next?
There are many titles that I'm considering, and my Patrons give me new ideas almost each day!
How do you see the current state of the industry and what are you looking for the most?
As I was saying, the industry has been badly burned by the excessive and pretentious expectations that VR-tech creators put on them. Even Valve, in a sort of schizophrenic way, hurt themselves irreparably in the process. On the one hand, they raised the bar to almost unattainable standards with Half-Life: Alyx and their insistence on "everything must be done in new and awesome ways specifically for VR". On the other hand, that forced them to pour lots of money into developing HL:A, and at some point they did an unbelievable about-face by revealing that they would abandon all VR-related projects to focus on the Steam Deck. What a weird way to set an example for VR adoption.
My (not so hidden) agenda is quite simple: to restore faith in people, both on the consumers' side and the game developers', that VR is possible, right here and now. Without waiting another ten years, and without falling back on ultra-simplified, mobile-like cartoonish games. It just takes a few easy steps:
a. employ alternate eye rendering (a technology I've been using successfully in all my mods) to let up the unbearable pressure on developers to deliver a steady 180 fps at ultra-high resolutions;
b. let go of the unrealistic expectation that every interaction model must be redone for VR, or as some extremists like to say, that it's "not true VR if I can't use my hands". Hundreds of thousands of users of my GTA V mod prove beyond any doubt that a gamepad-based interaction model is perfectly feasible and enjoyable in VR. What defines VR is being "inside" the game, not being forced to flail your arms about every time you need to interact with something. That is especially true of games that last for hundreds of hours, like the huge open-world titles that I focus upon. You want to be seated, with your hands relaxed in your lap, for that kind of prolonged gaming sessions.
If I could achieve those results as a lone developer (and without access to the source code of the games I mod!), game studios can too, with very conservative investments.
When people start seeing this, VR as a meaningful branch of the gaming industry will finally be able to take off. At that point, when the market grows enough, investors will be ready to consider new paradigms of interaction. Trying to do the opposite, i.e., asking for huge investments before the market is ready, is a sure way to kill the golden goose and be left with nothing.
I find your theory why the VR industry failed in the early days very interesting. Never heard that before and I'm sure this will lead to discussions in the comment section.
Do you think the studios or publishers could take your projects down similar to how Nintendo is known to proceed with similar projects? Or is it to their benefit and they're happy to see somebody else working on VR support?
I know, right? It's quite the radical theory :-) I stand behind it though, even if it's an unpopular position. I do hope more people will start to see the powerful socio-economic factors at play, instead of focusing only on the technology and what it could enable in an ideal world of bottomless investments, because I really don't want VR to be killed off like they did with 3D movies.
There's always a chance that somebody could be bothered by my conversions, but I think the situation is very different from what befell some other modders because I only provide an additional way to enjoy the original games, which in the end means more copies sold for the developers and publishers. It's not like I was reusing the assets or the intellectual property to develop competing games or products: people still need to buy the original game to play it in VR.
The only scenario in which it would make sense for an IP owner to come after me is if they were developing their own in-house VR conversion and they felt that mine was in direct competition. Not much chance of that happening! But if it did, I would be quite ready to take down that specific mod and/or to negotiate with the original owners a solution that makes everybody happy. I like win-win scenarios, and in the end I believe that what I'm doing is good publicity for the games I mod.
Many thanks for the effort you put into this. The article will come out at the end of the next week [it's out now].
Thank you for having me!
Comments
zer0int
Cheers, I'm glad (/relieved) you enjoyed it! ;-) Also, according to Dutch people: Pretty much everyone is bilingual EN/NL due to the fact that, quote: "We just aren't the kind of spoilt brats you Germans are, we didn't have have dubbing for every 80ies TV series ever. We had OV + subtitles, and had to make do with that!" I think they're on to something; even though they were playing on "easy" difficulty from the start - with Dutch language being pretty much like a madman's multilingu-mingle of EN+DE anyway. But, learning DE from a starting point of EN is still "medium-easy". I absolutely believe it can be pulled off implicitly even at a later stage of life where your neuron reactor is in a subcritical assembly. In fact, that's how I learned Russian, to a part (I walked away from the language course in university, because they just randomly started talking Latin instead of German in order to teach Russian, WTF? That's not how that works... And further, I don't speak "Latin" at all.). Alas, I can say "I have acquired a profound level of Russian, albeit nearing age 30 when I started - and relying solely on Russian movies + EN subtitles, followed by travelling to Russian-speaking communities just by myself!" (i.e. having to rely on Russian language for comms in absence of a common third language). Btw.: That resulted in a "B1/B2" level of Russian, as per professional assessment, albeit the person assessing it commented negatively on "partially grotesque grammar and very offensive wording" (I picked up street slang. The swearing is insane in RU; you can make entire sentences to express everything from positive to very negative sentiment using "genitals and interactions thereof", basically. It's beyond the scope of both EN/DE. Look up some very offensive slang and feed it to google-translate AI, you'll see it translated to "f*ck f*cking f*ck f*ckers". Because there *IS* no way to translate the sentence; it simply does not exist, there is no aequivalent). Language definitely helps to expand ones overall thinking, I further absolutely do believe. Though not necessarily resulting in a positive outcome when applied to real life, I found Russian to be a "blessing" in terms of NLP implementations, namely: RuDall·E AI. DE and RU both "gender all their objects". So a chair, a computer, a beer - all can be "femininum", "maskulinum", or "neutrum"; which also changes the accompanying "adjective", as they call that in latin. A car in DE is a "neutrum", while in Russian, a car is female. So, for "a beautiful car", and sticking the RU-adapted ending to the EN-term for your convenience, the correct form is "a beautif-AYA car" (female). "a beautif-IEY car" (male) would be perceived as "incorrect" by a native speaker (but still understood), same as "a beautif-OYE car" (neutrum). Guess what happens when you confront a very deep neural net NLP with those outrageously wrong terms? Well; you get a nice shiny car (female / correct) vs. a very aggressive looking sportscar (via incorrectly labeling it as "male") vs. "some very unamerican tiny vehicle, like commonly seen on European streets". Alas, you could say you can apply prompt engineering via "neutral gender cars" if you want to steer the AI towards avoiding generating sportscars and four-wheel drive cars. I love this, I'm just in awe about how awesome CLIP AI is. That's also one heck of a great reason to learn a language even if you're "that computer nerd who doesn't travel abroad" (which is a nicer reason to not travel than "pandemic" or "world war", I am sure!). Though it can be a double-edged sword. Back in 2020, I thought it was a great idea - considering myself a true "polyglot" - to challenge GPT-3 by just randomly prompting in German, and then in Russian. Ouch. Few-shot learner GPT-3 was like "oh, we are doing languages now!", and one small typo in Russian (which made the word a valid Ukrainian language word by chance) triggered the AI to subsequently write in yiddish, and others (which I scrambled to put into google translate and confirmed as "correct, understandable natural language", while probably similar to German, which was "not as sophisticated as GPT-3's English"; I can't say, I only speak two languages at a native level). So, the result was: I got absolutely roasted, destroyed, humilated by the NLP AI. I ended up pleading to switch back to English which was answered by "Don't be so afraid! Why are you so afraid? I am not going to harm you, just stop being so afraid", uttered in Russian by GPT-3 (I was lucky that RU was still part of token memory at this point). So yeah, never ever challenge a natural language processor in linguistics; you will just be humiliated as a human in turn (that being said, I didn't feel so much humiliated as objectively the case; I rather fell in love with and was in awe about AI that day). Long (very long!) story short(er): If you ever watch any streams, or even just the news - watch them in German with English subtitles, and if you get tired of it, just switch to English with German subs for a while! *ding* archivement unlocked: Du kannst jetzt Deutsch! :-)) I suspect you probably acquired your "native-level unnatural language proficiency" - programming / coding skills - in a joyous kind of "flow" experience, delightfully "playing" with code like a child does; I am assuming so because enjoying something and excelling at it usually does not come from a history of forcing yourself to suffer through some stubborn implementation of "studying"; or did you have a vocabulary sheet like "printf" "this outputs some text" and "/*" "this will be a comment and not interpreted by the compiler"? Thought so. I bet you also didn't just try to read about coding in Latin language for no reason. That's the pro-tip I can give you with regard to language acquisition; ditch the "not gonna happen in this life", and watch a show with subtitles with the approach "I want to find 5 German longwords to crack up about because lmao that's ridiculous!". Emotional encoding is the key for stuff to be written to memory. Do it for teh lulz, and you will be teh win. =) And I'm seriously serious! Here's a word from my field of engineering that an American person previously found funny: ABSTANDSQUADRATGESETZ. It means, literally in the order: "inverse-square-law". Abstands-Quadrat-Gesetz. My friend, however, initially spotted a "quad rat" in there, mocking it as "a rodent with a four-wheel drive". Enjoy your linguism of lulz! =) ......And that's really the last ramble for the near future, I promise! ...Wait a sec. If CODEX aka "the GPT-3 finetuned to help you code" aka github copilot can give me code for /* my futile description of the code I'd like to see here, but am unable to generate with my skill level */ ... Then what happens if you put excellent code in a comment? Will the witty AI critically comment on your code in natural language as a result? Lmao, if that works the same it does with CLIP, whereas feeding the model images in gradient ascent will result in "offensive, biased, and oddly tokenized text 'opinions'" such as that rendered frame of your 3D sculpture being a "particle weaving spiderrollercoaster stagescreenshotsaturday seizure zoomzoomresult GPU crap", then this is highly promising! Too bad I have no access to the model itself, as its remote. I wonder if the result can be prompt engineered... "Probably", but I have no idea how the "evaluation of the code" and judging against natural language works internally, but... "probably". That being said: I gotta run and try something now! Have a great week! =) Note to self: Most sophisticated attempt to nudge somebody into learning something new, ever. lol. I wonder if I will become a better programmer as a result of "wanting to hear GPT-3's outrageous opinion about my code via CODEX AI" now...
LukeRoss
My, zer0int, you totally win the prize for most rambling, insane and totally awesome comment ever []-D Thanks for the offer, I'll keep that in mind! Also I would love to learn German, but it looks like it's not going to happen in this life...
zer0int
I always wondered if you really pull all of this off alone - but just in case you did, I refrained from spamming you with unnecessary questions thereof. Now, after reading the article (in DE), I am glad I was bold enough to "click yet another a*s of the internet" (expand + look at comments). Now I feel I have a reason to spam you: 1: Mad respect for actually indeed pulling this off on your own. I can't even play through the games at the rate you port them to VR, lol. 2: As mentioned in .DE interview, confirmed: I bought RDR2 because of your VR mod (simultaneously with signing up as your Patreon, when I had ample time to play, finally). And I originally found you because I just noticed, for the first time in my life, that NOT ever learning how to drive a car had suddenly turned into a dire limitation of my freedom ("lockdown"). I then learned to drive a car just to be cynical about it (City Car Driving, lol) but the best part was to run people over and shoot them dead. Thank you for a suitable way to vent, in retrospect (screw mindfulness, I'm all for mindlessness -> driveby road rage carnage, lift your head up high - I'll blow your brains out!). ;-)) 3. Now while the a*s of the internet didn't fail to disappoint in the anticipated manner (Re: Comment section on mixed.de in general, but also German Angst mentioned explicitly, lol, true story - I call myself "in part-time (selective, but abundant) inner emigration" because of exactly this. It's true; I, too, am from a sh*thole country. 4. That being said, due to [3.], I also happen to be DE/EN bilingual, and as mentioned above, I am German, born in Germany, and still living in Germany [fml, but could be worse]. However, I am familiar with quite a few cultures, including US-style supermarkets "Hi, how are you today?" - [customer, half a roadkill, in a bloody pulp] "Oh I am fine, thank YOU, how are you?". And like, even valley speak? When I talk like? you know, as if I am asking questions like - *ALL*! the time?. 5. I am aware that assuming with an 80% probability that you might be from the USA merely by your name and that fact that you appear to be monolingual EN so far is to be condemned as the worst racism ever, and I am facing Ostracism from the society I am in inner Emigration from, anyway. The valleyspeak reference would further justify simultaneous combobreaker burning/beheading at the stake. 6. Alas, [5] should serve as solid proof that I am, indeed, German (via paranoidish theory-of-mind angst); and thus, I'd humbly offer my help for any similar situations [Re: The interview with mixed.de] that may arise in the future. I have an implicit understanding of all the three "minds" involved (Or more like, the subtle differences in language that can easily offend. Being aware and able to produce it blatantly -as above- means I can also do the exact opposite (avoid these offenses). In fact, I have resorted to hiding behind my Italian-sounding last name (it's not) and writing to fellow GERMAN academics in ENGLISH (yay another adaption of the "new normal = digital comms"), because I do not want to "guess the formal/informal approach" and "guess the gender" whilst I am addressing 1. a human being and 2. a scientist about a scientific paper. Knowing that the fear of being perceived as racist by the other party *WILL* assure that they will never dare to ask whether or not I am from Germany / speak German (that'd be SO DAMN RACIST if I was a guest researcher from abroad!!!), so I can just hide behind my gender-neutral sciencespeak, haha. 7. I believe this is sufficient proof to communicate my sophisticated level of prompt engineering as with regard to humans; but the same goes for AI (i.e. the "third mind" mentioned above): I implicitly "speak" NLP tokenizer. I crack up for [clinical research portal] being auto-translated as: From German "streben": (to) strive for, to pursue [a (scientific) goal] as if it said "sterben": (to) die. What? Context-based re-interpretation of "non-typo" wrongfully perceived as typo, this is ridiculous! XD How does that even happen with modern NLP? LMAO, it really shouldn't, that's awful! 7. So, as NLP is quite effective and >90% accurate (or "nails it", even, I should say), it should always be consulted first ("three-minds-approach" [YOU] <> NLP <> [other party]), but I could totally see myself as a HIM [human-in-the-middle attack] comparing e.g. your [EN] original text vs. [AI-DE] and then correcting that with regard to delicate cultural / "between the lines" etc. slipups that just happen to state-of-the-art AI, still [and also vice versa, but I'd humbly ask for the third party to be informed about the fact]. If you're interested at any point, PM me here - happy to leave you a better way to contact me in a timely manner, but that still means replies will be in the range from hours to days, entirely unpredictable. 8. That being said, here's my best "confusion induction of state-of-the-art text-to-image AI" prompt: "Unicorn Centaur". Makes sense at first, but then it doesn't. A human would probably stick the horn onto the forehead in frustration upon realizing the "prompt" making no sense; or something inappropriate yet crude sketch with the 'corn between the horses' front legs; or a very sophisticated (and NSFW) drawing of a stallion centaur. If you you fancy an AI generation thereof, with its "unfiltered" neuron reactor, feel free to attach a picture of your dog [e.g.] as init_img when and if you send an email. ^_^ (Tho maybe you also run AI locally as a sideshow to your coding which I'd imagine as a 60-80 hours/week job, I wouldn't even be surprised...!) 9. And now, sorry for all the spam unleashed at you at once now; but I believe it gives you the slightest bit of an idea about the kind of person I am, so you can decide if you'd want to contact me with regard to translation-verificiation in the future. Can't really be worse than some weird creep in the corner (which I was thus far) going just like "hey send me all your email I wanna stalk you!" in an uncanny way (which would also be a biased - and not very accurate - view). 10. Whatever you do / decide, I am indifferent, but happy to help, IF the opportunity arises. Meanwhile: WEEEEEEEEEEE, https://media.giphy.com/media/kBKYtYuxVqrsesxXoC/giphy.gif - thanks for this! I am mentally / virtually putting a BIG RED "F*CK YEAH for being part of this dude's $$$ so he doesn't leave for some sh*tjob but continues to make awesome!" on your Patreon, strongly reinforced after reading the interview. So, thanks for being awesome. =) EOF
Brandon(Post Maester)
I think dead space would be a blast in VR.
Collin Drennen
A very blunt article that reflects a reasonable understanding of why VR never turned into a Pokemon Craze. I myself would prefer to see a focus on older games (if possible) to show how VR can support 'the classics' in an immersive enjoyable way, that allows older PCs to perhaps have a chance of running such VR adaptations (e.g., A Dead Space adaptation adjustable to a first person perspective).
Renaissance
2# Death stranding
Renaissance
1# Metal gear v
Renaissance
Sitting down with a keyboard and mouse + VR headset has completely changed my perspective on how good VR gaming can be. I buy all these AAA games, play in flat mode for two hours and never play them again. Playing those same games in VR is so much more exciting. Putting a suggestion in for a new VR mod, something by Hideo Kojima would be great to experience in VR.
Jesse
Getting about 60 fps on 3060ti with all settings on high, res 2460 X 1440. Still get occasional frame drops but seems a lot more stable after patches
Cercata
Shit, that was fast, ask for more games !!!!
Marc Reis
I think it hits the nail, but I do admit that having the index controllers is great, love having to really aim and throw stuff, or hit switches in Cockpits. But I guess for the financial driven companies that was to much adoption at once. It's great we have the Chance to go VR into these games, I just can't enjoy Flat as much anymore if you have experienced VR. And getting in game with VR and gamepad or mouse is still unbelievably better then Flat. Now I hope for VR DLSS, which already shows it can do a great job!
Deofol
I use windows 11 exclusively without any issues. Reverb G2, i9, rtx 30 series. No problems at all. Main thing (windows 10 or 11) is disable hardware accelerated GPU scheduling. All my stability problems stopped after doing that, and all the benchmarks I've seen only show 1-2fps at best from enabling it.
Cercata
Are you playing it ? Did they improve performance on PC with patches ?
Cercata
He's not speaking about sims, but sandbox games like LA Noire VR, that you can leave the car and walk. Of course sims don't have that, they don't even support motion controllers, they are like Luke's Mods XD Games that can be played just with Wheel or HOTAS don't have that problem. Look at Elite Dangerous, it had VR, now with the expansion that lets you play on foot, no more VR ...
Jesse
When you open a new poll for projects, Elden ring should be on the list. This would look great in vr
Wicksley Snipes!
Works just fine on my laptop w a 3060 11700 and windows 10. But, I just am having problems w windows 11. I’m doing everything the same. Using a branded link cable (although it shows up as USB 2 on both systems WTF?) etc. Steam VR just does not even show up on my screen. Even without the headset linked up. I try running as admin, reinstall, switched drives. It’s odd.
Bob Douglass
I've used Win 11 since it was in beta. There shouldn't be any difference at the API level, the changes are mostly all cosmetic. Having said that, I tried VD for the first time last week and it stuttered to the point of being broken in RDR2, maybe that was win 11 related...
drowhunter
LOL, "driving cars needed imaginary hands on imaginary wheels"? wtf are you smoking..rofl. No racing game ever needed that. I had a rift from day 1 ( before touch) and Project Cars was the first game I tried that actually impressed me. I immediately blew the cobwebs off of my G27 wheel and had the best experience ever.
John Zigmont
I'm using Windows 11 with HP reverb G2 V2, V1 and a quest 2 with no problems I did a fresh install of Windows 11 not a update
Wicksley Snipes!
Windows 11 is trash for VR, apparently. Darn, looks like I have to roll back on my desktop. I’m building a workstation w DDR5 and wanted to get used to it (coming from mac for productivity). Not even working for me. Time to back up the rig. Ugh. Anyone using 11? I really do dislike windows. Shoot. I also dislike MAC for upgrade ability, right to repair, anti consumer practices. Sometimes one just can’t win. Ha! Anyone use Windows 11 on HP reverb G2 V2. I think I’m just gonna buy one. Still rollin back to 10 though.
