GameMakersToolkit
patreon
GMTK Game Jam 2020 Post Mortem
🕑 Added 2020-07-24 09:08:15 +0000 UTCHello everyone!
This month, I hosted the fourth annual GMTK Game Jam.
A jam, if you're unfamiliar, is a competition where people have to try and make a game in just 48 hours. And the game has to fit a theme, which is announced at the start of the jam.
If you're interested, you can now see my favourite games in this results video. Or go check out the full list of entries over on itch.
This year's jam was absolutely enormous. I knew the jam was going to be big when the sign-ups over on itch quickly surpassed the previous jams. By the time the event began, 18,326 people had joined: well over twice the number who joined up for 2019's jam.
Pretty much every year, about 30% of those people will actually submit a game (accounting for those in teams, people who forget, etc). And 2020 was no different. 29.8% of people submitted, leading to 5,477 games. Again, more than double last year's numbers.
These numbers actually put GMTK in a very nice place. Ludum Dare - a super famous game jam that's been running for 18 years - has always been bigger than GMTK. But this year, we snuck past them! Their latest jam in April received 10,000 sign-ups, and just shy of 5,000 games. (We're still far behind the in-person Global Game Jam, though, which saw 47,000 participants and 9,000 games in January!).
It's not a competition, of course. But it's still nice to be able to say that GMTK had - at the time of writing - the biggest ever online-only jam!
In this post, I want to talk about a few things that happened in the jam, as a way of remembering what went right and what went wrong: all with the aim of improving things for next year!
The big win: the new rating system
One of the biggest challenges with the jam is deciding who can rate the games. We've always opened the jam to public voting (instead of restricting it to just people who submitted a game) because I firmly believe that the jam should be as open and inclusive to all as possible. It should never be an insular event that's only for the hallowed few who can make games.
This does come with problems, of course. Namely that it's open to abuse and can slip into being a popularity contest. I was determined to find a solution though, and so asked itch if they'd consider making a system where people couldn't vote on whatever games they liked, but would instead be presented with a random game. They'd have to vote on that - before they got given another random game to rate.
Itch liked the idea so they ran with it, and gave it far more nuance. The final system (which itch finished making just a few hours before the jam ended!) presents public voters with 5 random games to rate. Each time they rate one, that slot is filled with another game. After rating 25 games in this fashion, the person is now able to vote on whatever games they like - giving them the same privileges as those who made a game (and never had to use the rating queue system).
I think it ended up working really well! It made it really hard to abuse the system because it discourages people from making fake accounts, or asking all their friends, fans, or followers to rate their game. It also distributed ratings to way more games: instead of everyone just playing the popular stuff, almost every single game got played and rated.
So in the end, we saw 143,005 ratings. That's a whopping three times as many as 2019.
99.7% of games received at least one rating, and the average number of ratings per game was 26.4: a huge increase from previous years.
Also, while you might think that the rating queue would reduce the number of public voters - it actually went up! In 2019, 40% of votes came from the public. In 2020, that rose to 46%. There also wasn't much negative feedback that I could see (though, some people thought the 25 threshold was a bit high). So all in all, that was a huge win.
The big fail: communication and rule enforcement
This one didn't go so well. When the jam began in 2017, it was a scrappy, upstart community-based event where I could quite easily handle the majority of the communication myself, make up rules on the fly, and answer each individual question. By now, though, it's just too big for that sort of approach.
At this point, I've got an incredible moderation team on Discord (thanks Slam, Adam, Sandra, Hobusu, and Brioche!) but they couldn't do their job properly without really clear, thought-through, and easily enforceable rules from my end. And I didn't always provide them. Which made their job impossible at times. Entirely my fault, sorry!
So, for example: while you can't change your game during the rating period, you can just upload a better version of the game to Dropbox and then write on your game page "The jam version is broken, go play and rate this one…" Is that allowed? Encouraged? Discouraged? Against the rules? Do we ask you to take it down, or immediately disqualify you? I don't know!
I need to think more carefully about what things will happen in the jam (we've done four of these now, so there shouldn't be too many surprises) and construct and communicate easy-to-enforce rules. And if a surprise does happen, I need a better way of handling that. I'm looking into making a website for the jam rules, so there's a single, easily-updatable, one-stop source for information.
Another thing that I need to fix is the grace period. After every jam, I've given people a certain amount of time to get late submissions to me. This year, I was utterly bombarded with over 300 emails from people. It took me and my fiancé until 1AM to sort through them all and reply, which just wasn't tenable. I've been working with itch on some solutions to make this easier to deal with.
Other lessons
Art and audio - This year I tried to reward artists and composers by having a presentation rating criteria, and letting people say if they made the art and audio during the 48 hour window.
Both were well received, but the latter needs more nuance (I'll change it from "all the art" to "the vast majority of the art"). Also, I'm working with itch to be able to set the weightings of each criteria, so presentation doesn't count as strongly towards the overall score as fun or originality.
(This year, people could personally decide on the overall score for each game. But it's also possible for itch to create an automatic overall score by averaging out the other results.
My influence - I do have to be mindful of my influence on the ratings. One of the winning games was "Cleaning the System", which I played on one of my streams, absolutely loved, and this caused a whole bunch of people to go rate the game highly and push it into the top 100. Good for me, but maybe not exactly fair.
This can also go the other way: on my streams, I was a bit dismissive of heavily narrative games (simply in relation to how they slow down the pace of a stream), which could bias the sort of games people make in future.
Obviously the best solution would be to not stream games at all. But because one of the main draws of this jam is that there's a chance that Mark Brown, king of video games, might play your game, I'm not going to stop doing that. I'll just be more mindful of my incredible powers.
Marketing week - I'm always a little concerned that the jam breaks down into 48 hours of making a game, and then… 160-odd hours of marketing it. And just in general, it's not ideal that people with a large fanbase (such as game devs with YouTube followings) will get more eyes on their game than anyone else.
The rating queue I discussed earlier definitely helped with that. I'm also not sure if itch should show the popular and top rated games during the voting week. Better to focus on random games and karma (which is where devs can increase their karma by voting on other people's games), perhaps?
But at the same time, I do think that it's apt that marketing plays a role in the jam. If a game jam is a microcosm of game development in general, then it's suitable that developers need to spend some time making a pretty thumbnail, getting their game to YouTubers and streamers, tweeting about their game, and so on.
So, definitely something to tweak but not something I want to remove entirely.
Some fun stats!
Okay, one final thing. Let's do some stats!
Pink River was definitely the most popular game in the jam. It received 866 ratings and 391 comments. Being on the front page of itch for a day probably helped with that…
Over 130 games were called "Out of Control", or some slight variation. Over 600 games had the word control in the name.
I asked people if they made their own art during the 48 hours. The vast majority of people (77.6%) did, which was interesting.
As for audio? Only half. This one's a little less useful, as we don't distinguish between music and sound effects. I know lots of people made their own music but used preexisting sound effects, so had to pick "no". That's getting fixed for 2021.
What about engines?
The majority of games were made in Unity, which wasn't surprising. I'll need to provide slightly different options next year: Construct and Stencyl weren't as popular as I thought, and there's a lot of games in "Other / None". So I'll break that down further next time!
Anyway, I think that's everything. I'll be seeking out feedback in a formal capacity next week by way of a survey, so keep an eye out for that. But if you have any questions I might be able to answer, please drop them into the comments down below.
Comments
Scott Cercy
Hey! I’m gonna reply to your reply here because, for whatever reason, it’s not giving me that option on your reply (I said ‘reply’ a lot). First, thank you for being so sweet and patient - you don’t get that a lot on the internet, and that usually makes me reticent to engage with people. So I appreciate that. Second, you’re absolutely right; I re-read your post here and it seems I misunderstood what you were saying. I misinterpreted “public” as non-devs, in spite of your clarification “whether they submit a game or not.” I thought that included people like me (I didn’t submit a game) when it didn’t (I did sign up for an itch.io account - although I didn’t sign up for the Jam itself, but I would, I just thought doing so was only for people submitting). Also, I originally felt that comments without votes would be “toothless,” if you will, without a vote; in other words, maybe the comments would be read but they have more of an effect when coupled with a vote. But I do understand that if anyone feels that’s the case for their comments, all they’re required to do is sign up for an account/the jam, and boom, they can vote. I get that. Especially since I agree, I can’t think of any other reason not to sign up other than to anonymously spam. You’re also right that I saw myself, and people like myself, as your target audience because I wasn’t separating the Jam from real life. It’s not as though the Jam is part of an instructional course that’s meant to prepare you for real world game releasing and therefore it’s testing you on each aspect. It’s just a fun competition. And I understand your point that devs are players, but as with anything else, we start to see the world the way we’ve trained ourselves, so there’s still a difference between someone who doesn’t make games and someone who does in how we experience games. However I think you’re right that this would more likely be the case if the devs were professionals, but since most participants aren’t, it’s unlikely this makes much of a difference. So touché again. I was coming from a mindset where people submitting games to the Jam were doing so as a way to prepare themselves for a career in game making, rather than just for fun or as a way to challenge themselves, etc. This was why I thought of you as making a game while “role playing” the mindset of someone creating a game for wide release - like practice basically. I get that that was narrow thinking on my part. So while this may still be a way of getting experience for a future career, you may not be wanting to think of all that peripheral stuff, you just want to focus on challenging yourself to make the best game you can. To that end, I can see why the opinions of other devs actually would be best at this stage (and additionally I appreciated your point that your true target audience is your own interest and originality so you can make your best idea) and why you wouldn’t want to worry about “marketing.” I was gonna touch on this a little in my last post, but I could see how it’s unfair that, in theory, someone with a less interesting game could get better results all because they spent time marketing it, while you spent time polishing your idea. I didn’t say this in my last post because, again, I was thinking like this should be real-world preparation and that’s a problem you may face in your career. Again, you’ve persuaded me: is this Jam meant to be a crash course helping devs to prepare for every aspect of the business or is it meant to be a fun way of seeing what clever things people can come up with in 48 hours? The Jam should have to pick a lane, so that you and other “submitters” can know this going in; do you spend the entire 48 hours polishing the game, or do you set aside some time for playing the business major? I think the queue helps with this a little (though yes, as you mentioned, people could spam through it); if people have to randomly vote on games, there isn’t as much of a need to focus on marketing, which frees you up to focus on the game itself. Anyways, I don’t know if I missed any other points you made, but either way, I just came to say that I recognize that I wasn’t combatting my impulse to be egocentric and was viewing your points through a very narrow lens, if not misunderstanding them entirely. I appreciate that you took the time to calmly engage with me, which, coincidentally, demonstrates that, if you can be this patient with a stranger on the internet, you probably don’t need much practice with customer engagement and PR, lol. Thank you for the great conversation!
Le Don
Hey, dude, I really appreciate your comment. Even though I have to disagrees with some points :). First off: You're not my target audience. I mean no harm, but honestly when I create a game for a jam, I make the game I want to make and try to make what I think might be the best version of it. And sure, I try to make it accessible and balance it (in one game I even added a little help for people with color blindness), but I don't make it for a specific audience in mind. And that has a simple reason: I know for a fact, when the jam is over and as long as I won't make it in the top 20 or 100 (which is very unlikely) and I'm not getting famous (also unlikely, but not giving up on that one yet), you or anyone else won't play my game. So the closest thing to a target audience would be fellow jammers, but I'm also not doing the game for them. Because that would mean, I don't make the game I want to make, but what I think other people would want. With other words, I would start to think about marketing and that would kill my creativity, which is the reason why I dislike marketing so much. I'm not wasting my free weekend for a project I don't have fun with. Having said that I want to make another thing clear: There isn't a difference between a game jam developer and a player. Game Jammers are also players just like you. Some of them are just 14, making their first game. Sure, maybe some of them work professionally, but they also joined to have fun. You mentioned games, that are loved by developers and not by the public... well, that doesn't happen in Game Jams – you can show me, if I'm wrong here, I'm genuinely interested, but I think you're mixing up Game Jam with the “real” industry. So having to talk with potential fans? I'm doing that with fellow jammers. Having an echo chamber? Here are my Ludum Dare games, I think the comments are fair and on the point (and never did I think "This is a player" or "This is a developer"): https://ldjam.com/users/le-don/games So that's the reason why I think your question about excluding “our voice” is – if I may say so – just weird. Firstly, as I said, there's isn't “our” or “your” voice. Secondly, when voting isn't public, you can still write comments – so your voice isn't silenced at all. Thirdly, you aren't excluded at all. I think it's fair when people, that joined the jam and didn't submit a game, can also vote. I'm just asking for them to sign up before hand, because forth: Public voting enables all kinds of cheating and also turns the game jam into a popularity contest. It's just not fair, when someone can get a boost by their followers. On the other side, I don't see what's elitist by asking to sign up before hand to be able to vote, which everyone can do. And here a small one: “I just want to make it clear that it isn’t fair to generalize and assume public voters don’t comment.” Which I didn't. I'm just pointing out that, if people are creating a new account to vote in favor of a game, they can just spam votes to bypass the 25 game-restriction. I hope that clears everything up. Have a nice day!
Erin Brioche
Happy to help! (Brioche here :p ) ^_^
Scott Cercy
I strongly disagree with Le Don’s opinion on public voting. I’m not a developer, I’m just a player - but that makes people like me your target audience in the real world. Just as there’s a role for marketing in the Jam (which makes sense), there should be room for the opinions of non-developers, specifically because they don’t have a stake in it. I don’t personally know any devs and I don’t follow devs on social media or YouTube, so my ratings and comments came as an audience member. Not only would excluding these voices seem elitist, but it would certainly skew the results as well. You could easily have a game that devs love but the public doesn’t care about; why exclude our voice? As for feedback, I provided context for every vote I gave (though, admittedly, I couldn’t vote much). I don’t mean to imply that I’m a representative sample, but I just want to make it clear that it isn’t fair to generalize and assume public voters don’t comment. I was incentivized by a) my desire to elaborate on my rating, itself encouraged by a sense of collaboration with actual developers - something I’ve never experienced; and b) though this doesn’t count for my first few comments, once I started receiving replies from devs, I was motivated to continue commenting. So in summary, excluding public voting could be seen as elitist, doing so would cause an echo-chamber, it deprives devs of the experience of learning how to communicate with their potential fans, and it limits the sense of fan-dev collaboration that the current setup promotes. Anyways, @Game Maker’s Toolkit (GMTK), this was my first year participating; I watched the results of the previous years, but I had never voted before. I had a lot of fun and the devs did an amazing job. I’d like to say that I really liked the random queue feature. As I mentioned above, I don’t know anyone, so I had no idea where to start (basically choice overload); the queue solved that problem. Personally I didn’t mind the 25-vote minimum. For context, I only got the chance to vote for like 30 games. I say this because for someone who’s voted for a hundred games, the first 25 can be a distant memory, but for me, the random queue comprised an overwhelming majority of my contribution, and I still wasn’t bothered by it. Of course, I can understand that people who are familiar with various devs may want to get to those games, so I empathize with that as well. I’m sorry I couldn’t get to more than 30; I’ll do much more next year (if public voting isn’t removed), because I loved the feeling of communicating with devs about their talents and passions. Even when I gave poor scores, the devs would reply to my comments positively and engage with me on my opinions. In a perfect world, there would be a subcategory of awards for devs who have good public relations skills, lol. Anyways, the developers are seriously talented people. The games were so much fun, and if this is what comes out of 48 hours, I’m excited for the future of the industry.
Le Don
Thank you very much for sharing this info and in general for organizing the jam. Just some comments on some of your points: - About the new rating system: I wasn't aware of the new system, but if people have to rate 25 games before being able to vote on whatever game, doesn't that encourage to just spam ratings? Since the first day I received 5 votes and only one comment and at the end I had like 16 votes and three comments. I always wondered why people were voting my game, but not commenting and figured, maybe they were just spamming for karma. Maybe I got more people to play my game with the new rating system, but in any case I still got no valuable feedback. - And in general I still don't get the idea to allow public voting. Because even if you only allow participants to vote (weather they submit a game or not), the GMTK Jam is still open to everyone. Everyone can play the games, comment on the itch-page and watch your stream. But why should they be allowed to vote games, especially as they don't have any stakes in it and don't have to vote for karma and no interest to leave feedback? You mentioned Pink River, which btw. is a great game, but one of the creators also has a Twitter following of over 4000 people and it was the most popular game since day 1. - I also don't get the approach to include marketing as part of the jam. The GMTK Jam is supposed to be about creativity, great design and such. But marketing is basically what kills creativity. It's the reason why Naughty Dog had to fight to have Ellie on the LoU-cover. Marketing is the reason, why you can have an awesome, well designed game, that no one plays because the art is bad, favoring artists above programmers. Just my personal experience from the last years, but comparing the GMTK Jam to Ludum Dare, LD feels like a little online festival, in which there are dozens of games to play and a nice community, leaving feedback to everyone. But GMTK is like the experience of submitting a game to an oversaturated plattform like Steam or the App Store and having it played by noone. Sorry for not being more positive.
Shadoninja
I completely agree with this suggestion. People will always be tweaking and fixing their game until the last minute. Some of those teams will inevitably break the game on accident. An unadvertised grace period is a great idea.
Shadoninja
One of my favorite Patreon posts ever. Thanks for the jam, care, and postmortem! As someone who sent you one of those 300 emails, thank you for accepting the late entry. Our functional build was submitted 4 minutes after the deadline and we were heartbroken at first!
Thanks for the great jam, Mark, and to all the devs reading this! This might be a silly question, but is there any chance to have the Out of Control logo as a wallpaper? I'd love a high-res version of it.
Keith Evans
Thanks for running the jam, Mark. My team and I enjoyed competing. There's a lot of feedback from our team regarding some of the rules. Code reuse being frowned upon but art assets being fair game is something we're fuzzy on, particularly for coders who build large reusable libraries to drag in to their projects. And while the new rating system looks like its getting more votes, it was weird as an entrant who didn't upload (another team member did) to also be asked to vote on twenty five other games first; doubly so as technically they are competitors. Perverse incentivisation is going to be impossible to avoid in all systems, but this one struck me as odd. I'd personally like to see how the ratings were distributed and whether there's a correlation between more extreme scores and the quantity of ratings a game received. That all sounds like a lot of complaining but overall I was impressed with the sheer quality of everyone's entries this year, and even though a lot of people gravitated to certain specific subgenres the expression within was all top notch. Fantastic jam, and looking forward to next year.
Arvin
For the grace period thing - I have a much better, simpler, and low tech solution. Just make the 'real' deadline 10-20 minutes later than the advertised deadline.
Virak
Awesome that the 25 random queue worked! I wonder if you could even separate those votes from non-random and see how much the ranking changed.
Sam Goodridge
It was so fun taking part for the first time! :) Thanks for your hard work! It was nice to take a break from my game engine project and do something free and fun! Definitely want to try out more game jams over the coming months. Ya got me hooked!
Tom Stuart
Thanks for sharing all this, Mark! It’s really interesting to peek behind the curtain. A couple of silly simple suggestions: * Just get rid of the idea of a grace period. It’s complicated and confusing and not worth the extra stress for you (and your fiancé!). After all, it means there’s still a hard deadline, it’s just at a different and less predictable time. I think it’d be totally reasonable to just say: here’s the deadline, sorry, rules is rules. There will always be someone who is unhappy about this but I bet there’s at least one person who was unhappy this year because their game was finished on time but then their dog ate their wifi router or whatever. You might as well accept the unhappiness and make things simpler and clearer. * In a similar vein, why create the “Mark’s influence” problem in the first place? The streams are really fun but there’s no law that says you have to do them while voting is still open. You could do them (live) after voting closes, or you could record them during voting but then premiere them after the close time. Maybe the latter loses the fun of the live chat but… it’s quite often just annoying and spammy anyway. So again it seems like there will always be some downside whatever you do, so you might as well put your foot down and eliminate stress & worry for yourself in the process. Anyway, brilliant jam, thanks for all the time and care you put into it. Can’t wait for next year’s!
Game Maker's Toolkit
Can I ask what reason you can't use WASD, is it an accessibility thing?
ZoidbergForPresident
"because I firmly believe that the jam should be as open and inclusive to all as possible" Well, 2/3 rds of the games I was forced to try were unplayable because of hardcoded wasd controls... so I abandonned my quest pretty fast unfortunately. A couple devs bothered to implement rebinds so khudos to them.
