Robc’s Blog

IndieGameTools.com is a new website that provides a means for evaluating game development tools.  Success of the site depends on providing valuable information to Independent Game Developers (Indies), earning their trust, and obtaining reciprocal feedback on the tools they already know and use.  As an Indie I already know what characteristics make a tool useful to me, but my needs, goals, likes, and dislikes may not be average, or even normal for that matter!  So before developing IndieGameTools.com I wanted to reach beyond personal experience to decide what characteristics make a tool useful.  During my research I came across an article Game Tools Tune-Up: Optimize Your Pipeline Through Usability by Dan Goodman on Gamasutra. 

In his remarks on applying usability testing results to ‘tune-up’ game development tools, Goodman references 10 statements (see figure) on tool usability and quality from a System Usability Scale (SUS) originally developed at Digital Equipment Corporation (DEC).  I was naturally attracted to the system for a whole host of reasons but in the end I decided not to use any of them.  I eventually settled on 6 simple, some might say shallow, attributes for rating game development tools.  I rejected the SUS statements because:

1) Even though the statements are yes/no questions, some of them urge a more detailed response through their nuance.  Since the questions are not simple, I believe that users will pause for just a moment and ask themself… what are they asking me?  That delay may be enough to cause very busy folks to skip the poll altogether (short attention spans + no time for pontification = no vote cast), and

2) I further concluded that there would be a high risk of some busy respondents misinterpreting the carefully worded questions and thereby lower the quality of the data that was gathered, and

3) Usability responses such as these seem more useful for comparing different versions of internal tools than evaluating commercial tools in which the respondent only has experience with 1 or at best a small number of the available products, and

4) The benefits derived from responses to these questions would accrue more to tool makers at the expense of time invested by tool users (benefiting tool makers is only a secondary goal of IndieGameTools.com), and

5)  I believe that a shallower approach to polling is a better way to build a knowledge base up from zero.  Perhaps after a gut level rating of the existing tools is in hand there will be value in asking more nuanced questions about a tool’s quality and usability.

Despite years of experience most of us really only have detailed knowledge of maybe a dozen commercial game development tools and those were likely chosen for us through our earliest job/school/contacts.  We simply can’t have invested the time to broadly evaluate these products and still made a living using them.  Successful developers do not have time to pontificate or to compare products, and asking them to do so invites responses based on speculation rather than experience.  Developers can however easily share their gut level answers to simple questions such as “I (could/could not) afford Tool X and it (was/was not) helpful, and the tool maker (was/was not) responsive to my needs. ”

Therefore the final list of attributes chosen for IndieGameTools.com were based on trying to capture gut level responses to similar questions (see figure).  The questions are not speculative, they require little thought, and have a small risk of misinterpretation.  Every vote cast in answer to these questions is likely to be more valuable than a vote cast under the auspices of a highly nuanced question.  So by skimming a tiny bit of shallow but high quality information from a lot of developers, aggregating it, and presenting it back to the development community, I believe that simple polls will prove very valuable.

I should make it clear that although I seem to reject the criteria and the value of usability testing for game development tools as recommended in Goodman’s article, I do so only contextually and on a very narrow basis when applied to Indie developers using this new website.  Where a team can afford to do usability testing of existing tools, I think the results will be valuable and Goodman’s recommendations stand uncontested.  In this case where we are trying to skim a small amount of highly valuable data from somewhat unwilling and unresponsive website visitors then easy to answer questions are best.

Before concluding let me say a few words about the goals of IndieGameTools.com in the context of all of the other websites and resources that are already dedicated to the indie community.  We learn about new products from marketing, independent reviews, and often through our professional networks.  IndieGameTools.com extends our professional network to include developers who have tried out products that we know little or nothing about.   This new website does not compete with existing professional associations, blogs, forums, or product review sites because we restrict the dinner table conversation to only a discusion of tools and tool attributes.  IndieGameTools.com is a point of reference to augment existing discussions in a thousand other forums all over the internet.  We provide html embeds that visitors can copy and use in their forum posts to point to particular products that they endorse (or perhaps hate) and we provide the community consensus of the worth of those products. 

So as the number of visitors to IndieGameTools.com grows and visitors leave behind comments and tidbits about particular tools.  The growing knowledge base will in the long run start to chip away at the desired nuances of tool quality as sought in the SUS survey statements.  Perhaps someday this website will offer similar quality information on the commercial tool market as can be obtained through big budget usability testing of proprietary development tools. 

-Robc