realvr
patreon
Gaming in VR at 60 fps
🕑 Added 2022-12-18 23:01:54 +0000 UTCIf you're here, reading this, it's likely that you already know how grandiose it is to play your favorite games in VR. Just to be there, inside the virtual world. To be able to crane your neck and peek behind a corner. Friend or foe? See and admire details that would never even register with your brain if you were playing on a flat screen.
And yet, if you're here reading this, it's also pretty likely that your favorite game is not in VR (at least officially), because publishers are so scared to invest in the sector, the poor things. I pity them. No, seriously, I do. I mean, it must be hard sitting on billions of dollars and giving up the opportunity of a lifetime, to be part of a revolution which is much greater than when movies transitioned from black and white to color, possibly deeper and more meaningful than the invention of movies themselves... just because you're so fixated on the bottom line that thinking of investing like 0.1% of your revenue only to build a market, without an immediate profit guarantee, has you running for the hills while mumbling "a fad... it's just a passing fad..."
But as you know, the number one item on my not-so-hidden agenda is "demolish any rationalization that can be used to resist the inevitable".
We already went a long way together with the rationalization "Aww, but I cannot possibly redesign my whole game to implement motion controls, they're so hard! Everything needs to be made grabbable and physics-responsive, all my costly animations for the thousands of actions that the main character can use to interact with the world will have to be redone or thrown away, don't even get me started on the nightmare which is to redesign the inventory, EVERYTHING in the game will need to be tested again..."
Ah, no. We proved that playing immersed, with exactly the same control scheme and interaction model as were built for the original 2D version, is perfectly feasible and greatly enjoyable.
What is the next wall of resistance that we need to break down?
Performance, or "you can select any frame rate for our game, as long as it's 60 fps"
When Oculus and Valve started the current iteration of VR, they boldly posited that every VR game should run at 90 fps. Well, that's kind of reasonable, methinks. Not so much harder than running at 60 fps as every other game does (erm, PC games at least: console versions still have a tendency to drag their feet at 30 fps).
But then, when the memo gets to the engineering team of a studio, their reply is that 90 fps is 50% more performance, or 33% shorter frame time. It takes months of work to shave down 5-10% of execution time from a well-optimized engine; 33% is a very big ask, they say. Also, since the time quantum in a Windows system is exactly 1 ms (Windows is not a true realtime operating system), and there are only 11 ms in a frame at 90 fps, there's really not much room left for maneuvering.
That's before you tell them that all of that will have to happen at the highest possible resolution that the engine supports, because everything in VR is life-sized and each pixel gets magnified and put under close scrutiny.
And before you tell them that actually there will be two full frames to render (one for each eye) in those meager 11 milliseconds of time. And no, don't even think of deep-pipelining the GPU, because latency is BAD!
As those of you who have been following me for some time know already well, I began to tackle this hairy issue since my GTA V mod by using Alternate Eye Rendering, which put simply means having the game draw only one eye in each frame time (so ideally in those 11 ms allowed by the headset refresh period). The other eye is reprojected, which means that the previous image for it gets rotated around the user's head in such a way as to simulate the relative orientation that the world would have had for the current frame when seen from the "stale" eye, if that eye view had in fact been drawn by the game engine.
That takes care of smoothing things out when you turn your physical head while in VR, and with a few extra engineering tricks (what you might have seen called "camera rotation compensation" or "yaw folding") you can also get a smooth rotation for the game camera with the gamepad right stick or mouse, as long as you're in first person.
But everything else is problematic. Strafing (whether it's your character that's doing it or other players or NPCs in the game), orbiting the camera around your avatar when playing in third person, moving your physical head from side to side, in general any kind of relative lateral motion between your eyes and the virtual objects you're looking at, all of that will trigger a psychovisual illusion caused by the rapid flashing of the low-persistency headset display, where it looks like the borders of objects get duplicated or triplicated. I simulated the effect, which happens in the viewer's brain so can't actually be captured on video, in my post from Oct 31: https://www.patreon.com/posts/74012973
This "ghosting" effect becomes worse as the game frame rate gets lower, because on average your vision will be hit by more flashes of the same repeated image.
If you manage to squeeze 90 fps from your game (drawing alternate eye views each time) the ghosting is barely noticeable, and you can get used to it and learn to "unsee" it in just a few minutes.
If the game runs at 70-80 fps, the ghosting is there but only annoying when NPCs move from side to side very close to you.
However (and that's the case for quite a few people), if your PC is not top-shelf and you can only get the game to run at the canonical 60 fps or even less, the ghosting starts to become an issue.
In fact, one question I get asked very frequently is: how come some native VR games render at around 60 fps and look very fluid with ASW for Oculus or motion smoothing for SteamVR, while your mods ghost like crazy when I hit 60 fps?
The difference, as Brian from Flat2VR never tires of explaining (thanks my friend <3) is that when native VR games, which are typically built using lower-complexity assets and dialing down quite a lot on the engine settings, say they're rendering at 60 fps, they mean 60 fps per eye, or 120 fps total. When a modded AAA game is rendering at 60 fps, it means 60 fps total, or 30 fps per eye. At 30 fps per eye, every existing reconstruction system (ASW, motion smoothing, etc.) breaks down completely and the game becomes unplayable.
Well, every existing system... until today.
AER v2
I can finally announce that I have my prototype working, and I'm already beginning to give it out in beta testing. Depending on how many problems are found in the beta, I will publish the first release next week or the one after that.
It's been a monumental endeavor (I lost track of the sleepless nights, it's all become a blur), and it still needs lots of TLC. In particular, two crucial points still need to be addressed. As I mentioned in one of my previous posts, the tech is conceived to be fully autonomous and adaptive. In its final version, it will be able to automatically adapt to the characteristics of the game, the system you're running it on, and possibly even to the complexity of the scene that you're playing at a given moment, and work out on its own the best configuration to apply dynamically to maintain the illusion of full frame rate within the current constraints.
I'm just one guy though, and implementing and testing all of that will take several more weeks or months, while instead I'm itching to go back to modding games for us all to play (yes, I'd also like to find some time to play my own mods every now and then!).
So I will be taking an incremental approach, releasing or re-releasing one title each time with the new tech customized for it, and gradually adding in the new features as they become available.
The first title, which already WorksForMe(TM), is Final Fantasy VII Remake Intergrade. With the upcoming re-release for this beautiful game, under the "Advanced" section of the "R.E.A.L. VR" tab of the mod overlay, you will have access to two new rendering modes. They are simply called "1/2 rate" and "1/3 rate". Apart from that, you will of course still be able to fall back on the good old Alternate Eye Rendering mode (which will now be called Legacy AER) and on Mono. Mono's mostly to keep Brandon's happy []-D
Except for possible coding errors on my part, the two legacy modes should behave absolutely identically to how they always did. It adds a small maintenance cost to me of course, keeping the old behaviors in place, but I don't like pulling out options and choices from under my users' feet, even though we're in 2022 and it seems that everybody operating in software does that systematically and shamelessly.
But let's talk about the new things!
The "1/2 rate" and "1/3 rate" modes will try to lock the game frame rate to the respective frequency, so for instance if you're using a Quest 2 set to refresh at 90 Hz (the recommended rate for my mods), which means 180 fps when running at full rate and drawing both eyes each frame, the "1/2 rate" will cap the game at 90 fps or 180/2, while "1/3 rate" will cap the game at 60 fps or 180/3. To make another example, if you have a Valve Index set to 120 Hz, the caps will be at 240/2=120 fps and 240/3=80 fps respectively (always remember that the starting rate is double the frequency of the headset, because the headset needs both eye views refreshed on each tick).
It is important to avoid stuttering or missed frames that no other external limiters are competing to also tell the game how fast it should render, so as always with my mods you should keep V-Sync off and not use any third-party limiters like RivaTuner. The internal fps limiter of the game is fine as long as it's set to a frequency above or equal the one you're targeting, so for FF7R it's all good because its frame limiter defaults to 120 Hz.
(Side note: a few people asked how to remove this limitation for FF7R. Don't do that. The game can't render faster than that anyway, especially at VR resolutions, and I observed that if you remove the 120 fps cap something in the driver or the game itself panics and hard-limits the game to 60 fps synced to the monitor, which is way worse than where you started from).
The "1/2" and "1/3" fractions are also important to keep in mind because they represent how many of the frames sent to the headset are actually rendered by the game, as opposed to those which are being synthesized (interpolated) by my new implementation. So, with 1/3 rate, each eye will get one frame rendered by the actual game engine camera, then two frames interpolated by my tech, and then again one frame rendered and so on.
So, which one should you choose? "1/2 rate", normally meaning 90 fps, or "1/3 rate" for 60 fps? The answer is yours to give, in the sense that it depends on the power of your system and on what balance you prefer to strike in the triangle clarity/fluidity/latency.
If you need your game to be super-responsive, and your system does not have lots of GPU power under the hood, then choose 1/2 rate and lower the resolution, or the game graphics options, until the mod reports a stable 90 fps. If you prefer nicer eye candy and don't mind a few more milliseconds of latency, choose "1/3 rate" and you will have a lot more headroom to move your graphics knobs. And of course, if you are the lucky owner of an RTX 4090, choose "1/2 rate" with all the bells and whistles and enjoy your perfectly smooth and insanely detailed VR :-D
The new paradigm ("give your GPU some breathing room") and what to expect
When the tech is functioning properly, basically every frame rate (90 fps with "1/2 rate", 60 fps with "1/3 rate", and further upcoming modes that will have even stronger interpolation capabilities) will look and feel as you were playing a native VR game, i.e., there will be no ghosting and no juddery motion whatsoever, because the headset will always be fed a steady 180 fps stream of images. That magic will last as long as you're hitting the stipulated target, and that can only occur reliably if you have some GPU headroom left.
In other words: if you ask too much from your system, and your GPU usage hits, say, 97% or higher, your poor graphics card will no longer be capable of completing all its required tasks on time. Visible effects will be: motion in the virtual world beginning to appear juddery or jittery; the runtime mirror window, or even the headset itself, starting to lose frames; tracking problems; frame drops in the capture if you're recording or streaming, and so on and so forth.
If that only happens for a second or two, no problem, as you might've just hit a rough spot in the game: let it pass and keep on truckin'. But if that occurs too often or even constantly, it indicates that you need to choose a more conservative setting on the interpolation scale. For now as I mentioned the mod will expose two, but in my secret lab under the [REDACTED] of the [REDACTED], I had acceptable results going as low as "1/6 rate". That means the unheard-of possibility of gaming in VR while pumping out as little as 30 fps total!
For the moment though you'll able to access a minimum of 60 fps, so if you want perfectly smooth interpolation and your system is unable to keep up a steady 60 fps you'll need to lower the resolution, close other apps, reduce your graphics settings and so on (*cough* upgrade *cough*) until you have a steady fps meter.
Incidentally: some aspect of my framework are long overdue for a restyling. One of those is the framerate counter, which is usable but nothing to write home about. One of these days (or nights) I will rework it into a visualization that can tell you more about what is causing you to miss or drop frames.
And to answer an important question that quite a few of you posted in the comments and personal messages during these weeks: the new AER v2 tech will NOT leverage DLSS3, for two reasons of importance and one incidental. The incidental one being, DLSS3 was not publicly available when I started working on this, a problem which could have been avoided just by waiting a bit.
The two important reasons are: a. I didn't want to restrict the benefits of the revision to only users with the super-new, super-expensive last-gen cards, and b. I wanted the two technologies to be usable together, so when new games start to come out that can barely hit 60 fps with DLSS3 switched on (it's going to happen, as sure as the sun rises, and sooner than you think) my tech can still be added to save them ;-)
Bonus
As if this post wasn't long enough yet. Told ya you would soon regret the days when I was being silently productive ;-)
The new releases, starting from the FF7R reissue with AER v2, will finally sport the magical slider you always dreamed of and never dared to ask (just kidding, many of you asked!): I worked hard on it these last few days, despite the brain fog, and it finally came together just this morning. All the reprojection/filtering/interpolation passes will have built-in automatic sharpening, which will be fully controllable by the user.
Again, to clarify. This is not just post-processing sharpening, which kinda works but can never quite restore the detail that was lost during previous operations. On the contrary, all texture manipulations will be done in such a way as to preserve all the detail in the original rendered images, and if requested with a sort of "overdrive" that allows for a sharper image than provided by the game itself.
Happy VR'ing!
(I apologize for all the unanswered comments and personal messages these days, I have been feeling really unwell. I seem to be getting better though, so I hope to be able to catch up soon!)
Comments
D S
For Luke: Can AER1/4 be enabled? I know you have reasons for not including it, but perhaps for us super-nerds? I've got a date with the Pimax Super and she's really picky about PPD. Thankfully she doesn't care about artifacts.
Matthew Murray
PLEASE PIN THIS POST! I've wasted hours and hours trying to optimize to reduce ghosting and artifacts in Ghosts of Tsushima, all could have been saved if I'd found this post sooner. This is ESSENTIAL information imo. Please make it easier to find đ
I currently think "Mono" works best for me, because it is the smoothest experience. In above text "Mono" rendering is not described. I know that my stereoscopic view is 30% of a standard reference. Many older games run on modern computers with a few hundred to a few thousand fps. For example try Quake 2 & do the benchmark on a 3090. Also, the Half-Life 2 mod could run with 3800x3800 pixels & the GPU was basically sleeping with 70Watt power drawn @ 90 Hz.
Would this be interesting stuff ? I for one, would easily invest in 2 graphic cards if it doubled the VR-show and see the best possible . Or is this totally unrelated to what you are doing ??
NVIDEA has some methods for using SL for VR https://www.youtube.com/watch?v=xaGK_ytNQfc and https://developer.nvidia.com/blog/vr-sli-accelerating-opengl-virtual-reality-multi-gpu-rendering/ and https://www.nvidia.com/en-us/geforce/forums/virtual-reality/11/244932/vr-sli-vr-sli-supported-games-list/
Update to using a PCIe 4.0 GPU in a PCIe 3.0 motherboard. Confirmed, my FPS nearly doubled. With the 1080ti, I was getting 30-50fps using low graphics settings. Now with the RTX 4070, I get 60-80FPS. Im happy with the results for the RTX 4070 @ $600, as there are other VR games that I could not push the graphics settings on. Now I can increase textures, without loosing to much FPS. Before, I had to sacrifice quality for performance. Plus, I can hold off and building a new VR system from the ground up. Now I just need to figure out the lockfree_table capacity exceeded and, now I get errors in RealVR64.log stating that there is a missing XR_SPACE_VELOCITY_ANGULAR_VALID_BIT. For now, Im dealing with every 5 minute to 1 hour crashes as I climb past 275 hours riding horses in VR. Giddy up!
Brandon(Post Maester)
HMMM, an interesting experiment! Let us know how it goes :)
Well, after doing some extensive reading. Came to the conclusion and advice per tomshardware, that "Game support for Multi-card SLI or CrossFire setups has basically died." So after reviewing another article by pcguide101, for "Will RTX 30 and 40 Work in PCIe 3.0?".... I purchased a replacement GPU for my 1080ti with an RTX 4070. Even though I'll be running it in PCIe 3.0, it should nearly double my FPS. I used both tomshardware "GPU Benchmarks and Hierarchy 2023: Graphics Cards Ranked" and the pcguide101 article to come to this conclusion. So, for those that need to nearly double their fps, but not ready to spend 1,000 on a video card to replace their 5 year old video card, will hopefully find this info helpful to make a decision.
I have the GTX 1080 Ti and I am still playing (REDACTED) game online using the R.E.A.L VR MOD v5.0.2, as its the only one that supports (REDACTED) game. I get a 'lockfree error' on start up, but I close at a certain point of loading. I play the game using Stereo AER for stereoscopic 3d viewing which provides the depth I need to feel immersed. Granted, it only works on low config, but I get gaming times anywhere from 15 minutes to 3 hours before it crashes the R.E.A.L VR. Easy enough, I re-start the R.E.A.L VR to hook back in and Im on my way. Even on low config, I can see impressive textures and can appreciate the art work that went into the game like the designer clocks and fireplaces. Any ways, I have 176 hours in the game, all using the RDR.Real VR MOD v5.0.2 and I love it! So, dont give up hope, some older systems can still work. For me, playing in pancake /flat-screen feels like my hands are tied behind me back. I only purchased the game and provided support here, because using the R.E.A.L VR MOD, is the only way I want to play the game!
LukeRoss
Mmm... I don't know, it's a pretty underpowered card for AAA VR gaming. Which game would you try on it? Please don't say CP2077 :-)
Trevor Erhes
Do you think gtx1060 can handle it?
Matt Evans
Youre talking about Praydog, and yes they work a little differently. Textures for instance in my opinion dont load quite as efficiently, especially if you have your texture setting lower like 1gb-3gb. He said when he developed the mods he was using a 2080, which seems a little underpowered considering whats being done. I have to assume he doesnt use AER either judging by the smoothness of motion, but I cant figure out how he gets the results he gets on a 2080 without using AER. My guess goes back to how his textures load and the fidelity of them.
Hm, all existing systems.... On my computer with new videoadapter Asus 4090 OC Resident Evil 2 with REFramework + mod with more detailed textures and all settings are maxed - works like a charm. No ghost effect. However FPS are around 40-60 there. But it works like I am phisycally there. So scary and so beautiful. The best game in VR I played (well, except Alyx). Did that guy do it differently?
