gamersnexus
patreon
Rearchitecting Testing for 2H17
🕑 Added 2017-08-08 04:46:23 +0000 UTC Hi all,Wanted to run a Patreon-only update!
We tend to build a new test plan once per year for each set of devices. The test plan covers data organization (a big, unseen element of what goes on behind the scenes), custom software development, test automation where possible, and determining which software is used for which products.
We have been on our current CPU testing suite for nearly a year now, so it's getting to about the point where we want to update. An update will allow us to take a year's worth of learning, of finding shortcomings or inconsistencies, and massively improve our testing for the next year. This is an iterative process, and is why our tests often slowly change with each new product launch: it is iterative and incremental -- those new tests don't happen overnight.
Sadly, this also means we'll be throwing away a year of data, as it'll likely be no longer applicable to our tests. We can keep some synthetics (like Cinebench), but will probably update Blender (thus losing Blender data), update Premiere, and change or add games.
Patrick and I have learned a lot about how we want to improve our CPU testing going forward. The first major step is to clean up the spreadsheets and data sorting and organization, then build-in some semi-automated chart generation. That should save me around 2 hours of rather tedious work per test. No need to manually generate that many charts for each review. We also want to add a few games, but we're not sure what to use yet. Possibly Civ6, maybe some others. Metro might be retired at some point soon, though we're not sure yet.
Anyway, the GPU test suite is still good for now. We're about to add at-rails power draw for RX Vega, which will be new as it's our first GPU review since getting the clamp, but will otherwise stick with the current list of games. Might add one or two slowly, and we're looking forward to Destiny 2 being a potential bench title.
The CPU test suite revisions will begin once Threadripper is fully out of our content pipeline, so that'll be a while yet. Once that's out, we'll start fresh with no data, sadly, and begin building a data-set as CPUs launch.
Cheers, everyone!
(PS - next round of shirts shipping this week)
Comments
GamersNexus
Sure! Expires in a day: <a href="https://discord.gg/yhkfaU" rel="nofollow noopener" target="_blank">https://discord.gg/yhkfaU</a>
GamersNexus
We might do just that. May end up developing the new one alongside running the old one.
Trausti Bergur Traustason
Looking forward to Threadripper coverage. Keep up the awesome work you guys do. Any chance of a new discord link?
Please, do not dump your testing after TR, as intel has announced the rest of its enthusiast line coming within the next month and a half. Maybe reconsider waiting till then if it will make for more comparisons and less work having to re-do TR testing.
True. Looking forward to whatever y'all bring to the table.
GamersNexus
We're aware of it and also think so, it just becomes a question of where we define "reasonable." Because we test a lot of other stuff (re-adding thermals in depth, just added current draw/power, added game streaming, already do production/Blender/etc.), it is difficult to assign more time for games. Presently, we stick to around ~5-6 games for our CPU benchmarks, and only use one GPU. We are looking into possibly adding a mid-range GPU as well, e.g. RX 580 (or RX Vega 56, something like that). But for the really games-heavy CPU benchmarking, we'll probably leave that to other sites. Too much to do with thermals/power/streaming/VR/production already, so only so much bandwidth left. We will be re-evaluating our games selection and trying to add maybe 1-2 more. The hope is to also automate more of the testing, which should free-up time to add another hardware configuration. In theory. We'll see :) Thanks for the support!
Regardless this kind of dedication to methodology is why I support y'all.
I don't know about everyone else, but I'm solidly of the opinion that cpu tests in gaming should be done with all (reasonable) possible permutations of hardware. Mainly referring to the tomb raider debacle back with the ryzen launch where green GPUs seemed to be underperforming on ryzen relative to Intel competition with green GPUs. Now that Vega is out I think it should be of utmost priority to test all CPUs with a gpu from each team. Especially for lower resolution testing methodology. I see no one else doing this personally. (Then again I don't really look too far)
