furarchiver
patreon
What's the deal with the extended downtime?
🕑 Added 2024-05-21 00:10:08 +0000 UTCYou may not know this, but Furarchiver actually crashes fairly frequently, often multiple times per day. You may not notice this because it immediately restarts, which leads to only one or two seconds of downtime, but it can still interrupt zip file downloads and let other requests hanging.
I don't know what exactly is causing this, and have been silently deleting the e-mails the error reporter sends me all the time. Furarchiver is written in C#, targeting .NET 6. If you mess something up in that environment and your program crashes, you get a nice stack trace (the chain of function calls) with source code file name and line numbers. This makes it fairly easy to diagnose and fix the problem. This is why the e-mail reporting exists (you can tell Windows Event Viewer to take actions on certain events), and it has helped me in the past to fairly quickly identify problems and fix them, but this one crashing issue I could not solve.
My problem is that it's not the furarchiver executable that crashes, but the .NET runtime itself, which I have no control over. All I get is a pretty useless error message about an internal error in the runtime.
The most likely cause is code that is not written in C#, but an "unmanaged" language (like C++ or assembly), which hints at a library that's in use. I suspect it's the library to access the database, so I decided to switch to a completely different database system. Furarchiver was previously running on MariaDB (a MySQL compatible db), and I've migrated it now to a Microsoft SQL server. Performance wise I expect this to be similar, but I hope it resolves the crashing issue. I also upgraded it to .NET 8, which is the latest long term version.
The data migration was successful, and the old database has already been deleted (after taking a backup of course). Furarchiver is still in lockdown mode. In this mode, non-admins are locked out of the application, but downloading of the queue continues to function. I will let the queue drain over the next few hours to see if there's an unreasonable number of broken files, which brings me to the next topic:
A new change I also introduced is for the archiver to store broken files. The artist downloader is frequently just trying to download broken files because that's eventually all that remains, so every time a person tries to gather files for an artist that was already archived, only broken files will be attempted again because all others are already present.
A file can be broken in two ways, it can be empty, or it can be incomplete. I detect broken images by attempting to pass them through the thumbnail function that the gallery viewer uses. If that fails, the file is assumed to be damaged. I also treat the "No Image" placeholder image from FA as being broken. It doesn't happens often and I don't know why it happens, but the backend for some reason does occasionally download just the placeholder.
Broken files are guaranteed to stay broken because the TOR backend service doesn't retries broken files. Because of that, they are now saved in the database too, so when an artist with broken files is scanned again, I don't have to retry downloading them. I only save broken files if they're older than 7 days to minimize the risk of the file being broken because I'm trying to download it from the backend while the backend is downloading the file itself from FA.
I've added a column to the database to easily identify broken files as such. This allows me to trivially filter them whenever you work with an artist.
Empty files will not show up in the gallery at all, damaged files will be labeled as such and replaced with a test image. When you download a zip file you now have the additional option to include broken files. If checked, they will be added to the zip file, and mentioned in the generated readme from the zip file.
As usual, if you have any questions feel free to comment here and I try to answer them.
