mastodon
patreon
Mastodon 3.0
🕑 Added 2019-10-19 13:37:19 +0000 UTCHello Patreon! I'm finally writing this. The release of 3.0 was somewhat stressful, because it was wrapping up a very long period of work. On one hand it's nice to be able to work on something for longer without interruptions, because it gives an opportunity to find bugs and usability issues as they come, change your mind about features, with less bureaucracy inbetween; but on the other hand it just builds up a lot of pressure to deliver, and more changes equals more bureaucracy in the end, as I have to transform a 563-long commit log into something a non-developer reader would find useful, and then select which of the changes deserve to be editorialized into a blog post. This time there's been so many new and changed features that the blog post had to be aggressively trimmed to the most impactful ones, and I'm still not 100% certain we selected the right ones to write about because the impact varies from person to person.
I pushed out v3.0.0 on October 3, but without announcing it on social media as the blog post wasn't ready yet (it needed some last-minute rewrites). The thinking was, make the code available as soon as possible, and the announcement can come later when we have a blog post that people could submit to link aggregator sites, at a more convenient time when link aggregator submissions tend to do better. Well, it didn't go as planned, as somebody simply submitted the long changelog on GitHub to Hacker News almost straight away. Can't complain about that outcome, though, as it got on the frontpage. Who knows if the blog post would've gotten the frontpage? I have a theory that people on Hacker News tend to click on familiar GitHub/Medium URLs more than on custom domains like blog.joinmastodon.org.
Unfortunately, just a day or so after that, Wasabi, the object storage provider used by my Mastodon server, mastodon.social--basically, the service that stores people's file uploads--started experiencing a major outage. So all the hype from new features, and new users from the news about them, were met by broken uploads, missing avatars, and general slowness. As the day progressed, things only got worse. What should've only been affecting file uploads was for some reason affecting the entire database, leading to ever increasing delays on users' home feeds. This led me on a bug hunt. Long story short, for those interested: The paperclip gem uploads files to storage within a database transaction (and okay, fair enough, can't blame that behaviour as you wouldn't want to save a row if the file didn't actually save). So if the storage (in our case, Wasabi) is slow as a snail, the transaction will be kept open for a long time. The long-running transactions were not so much locking up the database per-se, as simply exhausting the connection pool. But we tell it to time out after 5 seconds, so why were transactions being kept open for multiple minutes? Turns out the aws-s3 gem has a built-in, on-by-default, synchronous retry mechanism with exponential backoff. Turning that off and increasing the connection pool got us through, and it only cost a night and my sleep schedule.
Unfortunately Wasabi remained inoperational for like a whole week, if not more. I've experimented with using an S3 proxy service (flexify.io) to migrate mastodon.social to a different object storage provider, but discovered that with the amount of data we generate per hour (5 GB), the pricing of the proxy service (at 5 GB per hour, $1 USD per hour), and the fact that until Wasabi becomes operational the old data could not be moved to the new provider and the proxy service would continue to be necessary, I just could not afford it, so I unplugged it again. Thankfully, a day or two later, Wasabi finally came back online. Super stressful time though, and I am looking into changing providers long-term.
Anyway, by that point, I already had some fixes for bugs discovered in v3.0.0, and the blog post wasn't finished (due to illness, Eleanor could not work on it, and my hands were full with the Wasabi issues), so I decided to wrap up v3.0.1 while getting outside help for the blog post from Tremaine Fiske (professional copywriter). Finally, on October 11, both the blog post and v3.0.1 went live.
Still though there was so much uncovered ground left that I wrote an additional blog post just to explain how to use all of the new APIs and admin features, and then another one explaining how to add Mastodon sign-up to mobile apps--while not a 3.0 feature, I find that it is still severely lacking from current Mastodon apps. Then, I had a more or less spontaneous idea to record a "patch analysis" video--something people do for video game patches, especially multiplayer ones like Dota 2, where they go over changes one by one and discuss its meaning and impact--and hopped onto Twitch with Kurtis Rainbolt-Greene, where over the next 1h 15m I proceeded to gradually lose my voice. And we only covered one section of the changelog!
So here's where we're at now. I'm trying to get my thoughts together, focusing on non-development aspects of Mastodon. The joinmastodon.org server submissions backlog has been processed, the documentation is being re-worked with outside help, we're also still going to push a blog post out about Mastodon's now-possible use case in schools. We're looking into providing more merch options besides t-shirts, and might get another (video) ad for Mastodon. At the start of November I'm going to Athens for another Eunomia conference, which while also work, will at least let me switch up my environment for a bit after being holed up inside for 4 months.
Meanwhile, I would love to hear what my patrons want to see in Mastodon in the future. What would you like to see me work on in the next version?
Comments
Thank you for all of the hard work. It would be awesome to be able to contribute to this project. Maybe some day I will get there.
Petr Korolev
Thank you very much for your detailed feedback. I want to help, create and play around with my mastodon instance, and see what I can add on top. 0) Please let me know where a good entry point for the developer is? Is there internal documentation to get a brief understanding of how it works inside? Are there guidelines, how to run everything into the test environment that I can debug? (Probably there is some test instance with mocks of users/posts/databases?) 1) since you are focusing on the non-dev part, I would like to know if there is a chance to meet you in person at some conferences at the beginning of the next year? What are your plans? 2) from the feature parts: I will explore what we can do with privacy there. That is one of the most exciting topics for me. 3) I didn’t find any official mobile app in the repo. Just several closed source instances. I can help to make official ios/android apps, and it would be great to put it into mastodon github, so others can also reuse/improve this app. What do you think about this? 4) I’m wondering if there is a chance to make a universal login by the mnemonic secret phrase, just like cryptocurrency wallets work? It looks like we can bind our logins together, even without compromising privacy and security. I would love to implement it if you find it useful! 5) I covered a few points here for a start. Any suggestions where to move conversations about this? I believe GitHub issues would be a better place rather than discord. To preserve threads discussion. Or if there is any other option - let me know. Cheers, and thanks for everything that you do!
