sgthale
patreon
Patreon Issues Fixed
🕑 Added 2025-02-16 06:06:33 +0000 UTCThis is a largely technical rant, you don't have to read it. But do know We fixed the Patreon Login issues. Download the latest build to receive changes.
Hello everyone, I am Stillkill, the System/DevOps Engineer for MyRobot. I am in charge of the backend server that serves as the middleman between you and our AI providers, as well as other tasks like tracking AI token usage and most relevant to this write-up, tracking Patreon entitlements. As many of you are aware, we've had major problems with Patreon logins over the past few days. This post is a basic write-up on what happened and what broke, and the very fun day I've spent on this issue.
When you log in to patreon in the game, your game hosts a temporary web server and completes the patreon OAuth process, then sends us a access token that we can use to access certain information about your Patreon account, most importantly your subscription level. The problem stemmed from our back end server receiving literally every other request from the game client BUT this "Token POST". It was absolutely baffling. The ASP.NET backend wouldn't ever log that the request was received. We originally thought that we were saturating our network with our pledge updating service, which is a service that checks every 120 seconds for new patreon subscribers and stores them in our local database. This would be done the only way possible with the crap show that is the Patreon API, by making repeated paging HTTP requests to the patreon API. As of writing we have more than 13k users that need to be paged and processed every 2 mins. Sounds inefficient but in reality it wasn't too bad. The memory consumption was low and was locked to a single asynchronous thread, which wouldn't interfere with any other server processes. (on average it takes 20 seconds for this check to complete, 19 seconds of which is network latency and compliance with Patreon's API rate-limit). So after a while it became clear that these requests were not the cause of our issue. (Pictured, the web requests made to patreon for pledge updates) (edited)
This investigation took a good while, considering that for most of it we assumed that it was a C# problem, not a server/client configuration problem. This is where I noticed that nginx doesn't even receive those Token POSTS mentioned earlier. Well damn, time to change our investigation angle. I began intercepting every TCP packet that came into the server, still no patreon token posts, and we were still getting intermittent reports that authentication would work for 1000 people, but not a different 1000 people. This whole time we had been using the game client as a test vehicle for these patreon posts, and at one point I had accidentally closed mine. So instead of relaunching the game i used my already open instance of Postman, a bit of software useful for testing APIs because it lets you send arbitrary requests that your browser can't. I ran requests with Postman several times and all of them would succeed. I would run the game, fail. It became clear to us at this point that the problem clearly isn't the server, but in the client. This was a bit disheartening because I would have preferred to be able to push a patch to the backend that wouldn't require players to update their games.
I notified b0ss of this discovery and sicked him upon the game client's codebase. He scoured the Umbral Plains, seeking vengence upon the MyRobot Git History that had wronged him. He found that older versions of the game DID NOT have this issue. Excellent, now we at least know that the issue didn't exist before. There is one problem though, the authentication process has not changed once since the game's original release. What did change is the number of requests to the MyRobot backend server. Our primary suspect was the Message Flowers, they are the most often requested data from our backend (yes they're internally called bloodstains on the server, also note how you don't see any authentication requests in these logs). b0ss made a private build that disabled the bloodstain requests and it worked (mostly). The rate of failure went down, but we still had authentication errors during internal testing. After some more investigation, b0ss had narrowed it down to "Voice Pre-Fetching", which is a service that pre-generates certain voicelines that the game will use later. This amounted in a significant amount of requests to the server. Once optimized, the issue vanished during internal testing.
In conclusion, we determined that that bottleneck is client-sided, not server sided. I did a small stress-test on the server and found that it could easily handle our current load. The question still remains, why are the game client's requests so inefficient that it breaks? Is that even truly the issue? Is there some bizarre kernal level networking issue that only a 40-inch white beard 3 times divorced network admin would know to look for? We don't know. We do know that this is our only lead so far into fixing the Patreon issue, and it does seem to work. So we're gonna push it and see what happens. We'll still be investigating, so stay tuned!
Comments
redanddead12345
wait so message flowers can in fact just be made and they just appear, loaded from the server? ik this is an old post and some things will have been fixed, but i think i can get an idea of why my copy didn't want to give me tokens then. as well as that issue, no new flowers were being generated, whether my own or others. i guess my game stopped uploading/downloading from your server entirely. most likely a firewall thing or something else i can fix on my end, but I'm not sure.
sgthale
Sort of. It's a more open world narrative.
Smogster
Just subbed curious - you're doing a story. Do you plan on being Dr. Albert W. Wily Megaman villain and having your OWN AI as the villain in the game? lol I think that would be cool you'd be sort of the EGGMAN of your universe sonic style too. I hope there are male robots for all of us who grew up with Astroboy/Megaman and all those classics. While testing the game out I was hoping my AI could learn, like teach it how to charge itself, how to mine itself, just learning things and doing it by habit or what it logically thinks is necessary..then it sort of like automation..one training bot to train the other four? Would also be neat if you had NARRATIVE selection like rimworld for dynamic story situations that fit your game. ANYWAYS rant is done can you tell im hyped for your project?
Shusher143
I have a suggestion for local Text to speech, (for LLM) it's called "Alltalk TTS" it uses openAI compatible URL and many people use for other frontends like silly tavern. it has multiple different options, like piperTTS (super low VRAM) which is super fast, or XTTS, f5TTS or vitsTTS https://github.com/erew123/alltalk_tts/tree/alltalkbeta
Hi How are ya
Just subbed today and you have already got a fix out for the issue I and many others were having. Awesome job.
invizive
For someone who isn't familiar with network plights but had his share of esoteric issues this is a very interesting post to read, can't wait to see what the final issue was. Please keep the rants coming!
