Findings and Discussion
User Testing Alpha and Beta
User testing for alpha and beta was conducted at the Urban Future Lab, a cleantech incubator at NYU, and the Leslie eLab, a resource center for student entrepreneurs at NYU. Users were asked to go through the site, talk about what information would be helpful, and what information was missing from the page to make it a valuable resource. Feedback at this stage informed decisions like decreasing the size of the header graphic to give users access to the data more succinctly. The layout of the company page elements became more compact with every iteration and led to an increase of user satisfaction. The layout itself was iterated and tested multiple times to understand which worked best for the information that was displayed.
User Testing Gold I
At the gold stage of development, Usertesting.com provided a scholarship for the project. The first user test involved 48 users, 16 on desktops, 16 on tablets, and 16 on mobile devices. Of the 48 participants, 91% would use the website if they were to start their own company. 98% agreed that the website was easy to use and 70% agreed that the iconography used on the website was funny. As far as the overall rating for the site, 29% rated it 5 stars, 44% rated it 4 stars, 19% rated it 3 stars, and 8% rated it 1 star. Of the users that rated the overall website 3 or less stars, the most common comments were that they wanted more information regarding reasons for failure, information about the company or business plans. Of the 48 participants, 47 did not know of websites similar to this one. The one participant that did referenced CBInsight’s Post-Mortem site and preferred the graveyard.
In response to user testing, the goal of the project shifted from showing as many companies as possible to showcasing less companies but providing additional details through clear communication regarding the reasons for failure to create a more satisfying user experience. Additionally, many users tried to click on the reasons for failure, which were not hyperlinked. This revealed an opportunity to link these icons to other companies that displayed the same icons through meta tags. Participants were asked to verbalize their thoughts as they performed the following tasks:
- Task 1: Look around the home page and talk about what you think the site is about: what can you do here, what's it for, what strikes you about it? This task was designed to be open ended to examine how long it took for a user to understand the goal of the website and identify what elements are most striking without eliciting subjective bias. This task took users longer than intended, and text was added to introduce the page to resolve this issue.
- Task 2: Pick a company. Read through the page and talk about what you think about the content and the design. What stands out most? This task revealed more context and users began to understand the purpose of the website. The reasons for failure stood out most to the users. In verbalizing their thoughts about the content, there was confusion regarding some iconography used to present objective information, specifically:
- Founders. Those unfamiliar with startup culture did not understand what a rocket icon meant. The text “Founded by” was added before the founder’s names to solve for this problem.
- The amount of money raised. This was mistaken with income the company made. The text “Raised” was added before the amount of money to solve for this problem.
- The number of rounds. Those unfamiliar with startup culture did not know the term “rounds” signified rounds of fundraising.
- What “investors” meant. Confusion here was preceded by confusion regarding what the money signified. The text “Backed by” was added before the number of investors.
- Task 3: What are the benefits of visiting this site? At this point in the user test, almost all participants verbalized that this would be an effective resource for those looking to start or have recently started their own company.
- Task 4: Navigate to another company. How helpful would the information on the page be for someone starting a company? Most responses were positive. However, some users expressed the need for additional information, some of which is not publically available (ex: business plans for each company).
- Task 5: Do you know any companies or websites that are similar to this one? If so, explain how this one compares? One of 48 referenced CB Insight’s Post-mortem blog post.
Overall, the feedback regarding the content and design was positive. Having the ability to witness user interactions to the coffin designs and animations reinforced the slightly humorous tone of the website.
User Testing Gold II
The second user test involved 18 users, 6 tests for desktop, 6 for tablet and 6 for mobile devices. The same tasks were given and metrics were collected (Figure 5B). To narrow into the target users for Startup Graveyard, participants were required to agree to one of the following statements:
- I have tried to start a company in the past
- I want to start a company in the future
- I have successfully started my own company
Of the 16 participants, 88% agreed they would use the website if they were to start their own company. All participants agreed that the website was easy to use and 88% agreed that the iconography used on the website was funny. As for the overall rating for the site, 33% rated it 5 stars, 44% rated it 4 stars, 17% rated it 3 stars, and 6% rated it 1 star. Of the 18 participants, 17 did not know of websites similar to this one. The one participant that did referenced TechCrunch’s Deadpool site and preferred the graveyard.
This test incorporated a new metric to measure whether it was clear what the website was for, and 88% of the participants agreed that the goal of the site was clear. This was reflective in their reaction to the new introductory text on the website and in the decrease in the time it took them to complete the test. This could have also been impacted by the participants being filtered through the screener question.
Deciphering what measures could be used to identify failure in an article or blog post was a challenge. Patterns emerged as more articles were analyzed, but decisions to identify the reasons for failure were made very carefully. There were cases where founders identified a reason like funding, which is usually a symptom of a bigger problem. Funding was only included in cases where the founder made a good case for it. It is difficult to point to a reason and claim that a startup failed because of it, companies are very complex and their downfall is the result of interrelated long term factors. For this reason, the following text was added to the footer of the website: “Note from the undertakers: As you explore these grounds, be advised that though this site is meant to be used as a resource, it is not as a source of absolute fact. Companies are very complex and their journey here is usually the result of interrelated long-term factors.”
Another challenge was that users asked for additional information like business plans that were not publically available. Though there are a few companies that are open with this type of information, most companies do not make this information public. Poliana is an example of a company that open sourced their code and design assets (Poliana 2016). Wattage is another example, they provided a link to their well received pitch deck for download (Wattage 2016).