Why are there so many bugs?!?!?

Sharing is caring -Tweet about this on Twitter0Email this to someoneShare on Facebook0Share on LinkedIn0Share on Google+0

Ever since I’ve started coaching people through building their first (or 100th) software product from newbs to established software companies I can’t count the amount of times I’ve heard “Why are there so many #%$@# bugs!?!?!?”.

As a software people, we need to understand that bugs are part of the process.

We also need to understand that it is not the developers job to find bugs. They’re busy building your product and are only thinking of the use cases that they designed.  So when they test, they do not catch much. Their time is also better served building your product.

If you’ve left testing to your developer you’ve probably came to the realization that you need to take this into your own hands and test yourself.

This is great at first.Until you put out a few builds and find yourself doing the same thing over and over and over.  You hate it.

Then you release to your first customer and you realize that not only do you hate it but you suck at it too.

Don’t worry. There is hope. There are awesome people out there that test for a living and know an efficient way to catch bugs so your developer can fix them before they get to your customers. In this post, I’m gonna show you how to hire an inexpensive but effective QA tester that will save you time, money, and limit the bugs that get to your customers.

Job Post

Throw the following job post up in ODesk, Elance, or your freelance site of choice (I love Odesk).  Tag the post with QA, Regression testing, and other relevant skills. Then use the job description below (after you edit as necessary for your needs). NOTE: The overall template for this job post was developed by my good friend Amit who runs Outsourcing College.  Years of testing went into this template so do not reinvent the wheel.  And if you want to learn more about outsourcing signup for his list and attend a free webinar.

JOB DESCRIPTION_______________________________________
We need an amazing QA Tester to test our software product, log issues, write new test scripts and produce results on their own with minimal direction.
1) You will be testing our web application, logging bugs on GIT, and posting results.
2) You will be writing test plans and test cases as we build new features.
3) You will be assisting when we release new versions of our software – so we will often ask for your help with short notice and expect a quick turn around.

1) PROVEN history of testing web applications
2) Experience Testing with multiple browsers (Chrome, Firefox, Safari, IE)
3) Extremely Detail Oriented
4) Solid understanding of how to write test cases.
5) Problem solver. Organizational skills are good to have.
6) Proactive in giving status and very fast to respond
7) Your hours must overlap with 7am to 9pm Mon-Fri GMT-8 time (ie, USA Pacific time. For details check out http://www.time.gov/timezone.cgi?Pacific/d/-8)
8) Must be available on skype or gchat during those hours
9) Must be fluent in spoken and written english – must be able to talk via Skype or google hangouts (with voice)

YOUR BENEFITS____________________________________________________
1) We hire for LONG TERM.
2) Great team with friendly people.
3) Several of our team members have made over $1000.
4) We help you succeed at your task.

HOW TO APPLY_________________________________________________
1. Give short explanation on why you think we should hire you instead of anyone else?
2. Give at least 1 example of a job you have had testing web applications. What was your role? What did you do day to day?
3. Please also include an answer for this pre-interview question: If someone put a gun to your head and you had to create a process to test a new web application that you have never seen before what would you do? It should include how you write test cases, log bugs, and work with our remote team.
4. All applications must include the keyword “cool giraffe” at the top.
5. No bids above $3.33 per hour will be accepted. Lower bids give you an advantage. Please Bid appropriately. You bid rate may get you rejected. The lower the bid the better.

Interview Questions

Ask the following interview questions to your top candidates via email or skype (I start with email then do skype after). Choose them by looking at their background, reviews, past jobs, and question in the post.

What is a defect? When writing a defect what is included?

Have you worked on a brand new product before? What was your role on the team, tell me about how the QA process worked for that product?

How do you react when requirements are not clear?

How do you know when to stop testing?

Test Assignment

Choose you’re favorite one or two based on the answers above and ask them to complete a test assignment. For a test assignment, have them test a specific part of your app, then write test cases based on some loose requirements on how it should work. Also have them document any bugs they find along the process.  When I write up a test assignment, I always write some instruction incorrectly to see if they catch it. For example in the one below I give the incorrect calculation for one of the ratios.

You learn a ton about them from this simple little test:

  • Are they detail oriented enough?
  • Do they follow directions?
  • Do they include everything that should be included when writing a defect (build, browser, steps etc).
  • Do they cover everything with their test cases?

From this you will easily know who to hire.

Check out my example test assignment below.


Here is what I’d like you to do for your first test assignment. The assignment is for 4 hours.

Please take up to 1 hour to familiarize yourself with the product and how it works. Go through the few things outlined below to get comfortable.

If anything is ever unclear do not feel bad about asking questions. Never feel like you should know the answer… Any question is a good one… In completing this assignment I am going to be looking at 3 things: 1 – How well you perform with minimal direction and minimal requirements, 2- your knowledge of the QA process, 3 – How well you follow instructions.

Note: this application only works in Chrome and Firefox so far.

Start by creating your own account:
1) Go to yourwebsite.com
2) Enter name, email, PW
3) Land on /getstarted
4) Enter: Company name, phone number, credit card number, CVC (for credit card you can enter 4242 4242 4242 4242 as the credit card number, 01/2014 for expiration date, 123 for CVC) This is a test credit card…
5) Land on settings page

From here you can create some clinics, and add therapists to those clinics so you can start entering data.

Some information on how the calculations work in the dashboard.

When you are in the front desk – make sure you select patient visits in the drop down (screenshot – http://screencast.com/t/tBk08sfsCg0)

Patient Visits = Sum of “Total” in the front desk for the chosen time period
New Patients = Sum of “Initial” in the front desk for the chosen time period
Arrival Rate = Total / Scheduled
Visits per Eval = Initial / Total

Watch this video for a more detailed description:

Once you are comfortable, please complete the following:

1) Write some test cases about how the calculation of everything on the dashboard should work according to what I showed you and based on your own knowledge. Do this in the spreadsheet I started here: http://bit.ly/18Uay6p. Add any additional columns and change this sheet however you think it would work best or anything else you think is important to capture in a test case.
2) Explain in a paragraph how you would plan to make the process of regression testing the calculations on the dashboard as quick, thorough, and accurate as possible.
3) Log any defects you find. For now do this in the defects tab of the same spreadsheet (when all goes well Ill add you to our bug tracking tool. Include all the details you think are important to include in a defect. (Hint: the more details, the better).
4) Bonus – If you have extra time, click on the reports tab and create a report. Log any defect you find when creating a report. Here is a video of me walking through reports: http://screencast.com/t/aYOvv6gSdRg

Thanks and I can not wait to see your awesome work! –Carl

First assignment after hire

Once you bring someone on you should give them access to the tools they need.  I use GitHub for bugs, the GitHub wiki for processes, and a google spreadsheet for test cases.  In my case, aside from that all he needed was access to my application and instructions. Below is what my first assignment looked like.  Feel free to pirate it if you want (it will make more sense if you do use GitHub for bug tracking).

Here is what I’d like you to do.

1) Enter the defects you found and entered into the spreadsheet during your test into Github. Please be sure to tag them with the appropriate tags and is in the appropriate format.

For example:
Defect Title: Reports – For reports weeks start sunday, everywhere else is monday… Causes confusion… (always begin the defect with the area of the application it affects as shown.

– Tag the application with the relevant area of the application (Reports)
– Tag with the type of issue – Bug (almost always this for you)
– Tag the issue with priority: P1 Major issue, P2 Medium, P3 Minor.

Be sure to enter the other important details like OS, Browser, Environment into the body of the defect. Along with the steps to reproduce and an explanation of the issue.

2) Continue writing test cases to try to get 100% coverage of the application.

Please spend 4 hours on your own then checkin with me and let me know what you have done so I can check it out.

Moving forward

From here you have an amazing start.  You’ve hired a solid QA person that knows how to do the job.  You’ve given them their first task and you will start seeing bugs come in without you doing more that writing a few instructions by email or on a jing video (my personal favorite). If you want to see what good test cases look like you can see a copy of mine here.

From here refine your processes by asking your developer(s) “what can QA do to make defects clearer for you?”.  Send that on to QA.  Then ask them to add that to their procedures.  Always have your QA person documenting the process they are using and every piece of feedback they get from you. We are starting to use the wiki in Git for that. For example: Regression Testing, Test Case Writing, Defect Writing, Defect Process, etc.

Hope this helps you release awesome software with less stress 🙂


PS – If you’d like to learn more about the development processes and other things I mentioned in here, let me know in the comments.

Sharing is caring -Tweet about this on Twitter0Email this to someoneShare on Facebook0Share on LinkedIn0Share on Google+0
  • Jeff Fleming

    Do you or does Amit have a template for hiring the original developer(s), along with some tips to help get the “A” players that Dane makes reference to?

    • Carl

      Hey Jeff,

      I do have several methods for finding developers. I focus on the rockstar method because I simply don’t like doing the extra work and risks associated with hiring devs over ODesk.

      Occasionally I will hire a dev on ODesk but only for simple, not so important tasks. Never to work on my SaaS product.

      Happy to do a post on ways to find rockstar devs or technical co-founders.


      • Lyla Anne

        That would be so awesome if you could do a post about find rock star developers!

        Also, I’m curious about contracts. Do we need to have contracts with our developers, so idea’s aren’t stolen or to ensure that work is properly done? This may be a newbie question, but do we pay them upfront or after work is completed?

        I sincerely appreciate your time in answering these questions.



        • Carl

          Sounds good Lyla!

      • Nate

        I’ll put my hand up and say that I would love to hear about ways to find rockstar devs.

        • Carl

          Cool Nate. Will do a post on this too.

          • Patrick Clifford

            Great! Thanks Carl.

  • Kai

    Great post Carl! And I have too join the crowd here and agree that if you can do a post how to find rockstar developers that would be amazing!

    Thanks for doing this all of this!

  • http://www.pukkacrush.com JackMSVaughan

    Carl man, this is really useful thank you. I’m applying again for the Foundation & this is an area I think I’l have massive growth in. Good to get started on material like this asap!

    • Carl

      Right on man.