Software testing will be one of the most sought after professions well into the future. This is an interesting time to be in the field of software testing and quality assurance. We represent the eyes of the customer and it is our responsibility to ensure we are there to identify risks and report deviation. Exploratory testing is one way to do this.
There are two excellent books that can help here: “Lessons Learned in Software Testing” by Cem Kraner, James Bach, and Bret Pettichors, and “Explore It” by Elisabeth Hendrickson. Based on the insights gleaned from both the books, as well as my personal experience, here are few major things we should consider when planning to go for exploratory testing.
As testers we should understand that we are not the only ones responsible for quality. It requires the entire team to take ownership. As testers our job is to ensure that within the time provided for testing we find the maximum number of defects. We should investigate the actual behavior of the system and document anomalies. QA teams are in the business of communicating risk. As a QA team we should explore products, document our explorations and share our results with stakeholders to ultimately make informed decisions.
Session Based Testing
The definition of quality is the value given to the customer based on the requirements provided. In this context, the value that is delivered is defined by the customer. Exploratory testing was coined by Cem Kaner in 1988. It involves simultaneous learning, design and execution. Another big component is chartering which has been delved upon in detail in a section below.
The definition of testing completed for a feature is that it is checked and explored. Taking time to go in and do end-to-end exploratory testing will help uncover unexpected behavior. Exploratory testing gives full freedom to the tester to think creatively and identify all issues related to a particular behavior without having a pre-defined script to execute.
Session based testing is documenting the effort of exploratory testing. A lot of times, exploratory testing is confused with adhoc testing but the difference is that – in exploratory testing there is a formalized mechanism to document the findings. This documentation of results is achieved with session based testing. The first step in a session based testing is chartering.
Chartering is essentially setting a clear mission for the session to be tested. As a tester, the job is to be an investigator and ask questions about the user story as well as define the scope of the mission.
One example of a charter is “Explore the login field with security hacks to discover vulnerability.” The session can be established using the Pomodoro technique – take the test and break into time boxes. Focus on the test for 90 minutes. Turn off your email, social media and everything else and just focus on the session. At the end of the session the testers should take five minutes to reflect. If the team works in pairs while performing session based testing it is found to be more effective.
Here is a simple charter template to follow:
- Explore – mention the target
- With resources
- To discover — fill information
The “target” is what you are exploring. It could be a feature, requirement or module.
Resources can be a dataset, technique, or independent feature.
Information – What are you planning to find? Are you looking for any violation of standard or are you characterizing performance, security, functionality, etc.?
Another key aspect to consider while performing exploratory testing is to Identify the “never” and “always” conditions. Work with your stakeholders to make a list of things that ‘should never happen’ and ‘should always happen’.
Debriefing is key to exploratory testing: How did it go? What were the risks? This will help you to log defects appropriately.
Benefits of Exploratory and Session Based Testing
Exploratory testing allows the tester complete freedom to explore the entire functionality in question without pre-scripted test cases.
From a scientific perspective the charter ensures that your left hemisphere of the brain, which relates to logic, is taken care of and enables to free up the right brain to think creatively and come up with new ways to test.
Based on our experience, of executing exploratory and session based testing, we have found issues for some stories/functionality. We would not have come across such issues in the absence of predetermined test scripts which are usually based only on the acceptance criteria.
Once the defects are documented, it helps in identifying gaps in our existing test repository which can now be updated based on the new information, resulting in greater coverage.
We can also use heuristics to test some examples. A heuristic is a rule of thumb or educated guess. Some of the ones mentioned in the book “Lessons Learned from Software Testing” are mentioned below with my take on each.
- Test important tests first: Prioritize the most risky items first. You can’t test everything but you can always test as many as possible.
- Focus on most important risk first: This is of course risk-based testing.
- Avoid over scripting: This is the key to session based testing as well. We want the testers to think creatively and out of the box. You can’t be confined totally to just the script.
- Implied requirements: Testers usually test against explicit requirements and acceptance criteria. The key is to go beyond that and test implied requirements as well. We need to keep the end user in perspective in our view to complete this activity.
- Collaborate with dev teams: Quality is the responsibility of the whole team.
- Rapid feedback: There should be constant communication between the dev team and QA team. The testing should be designed to provide immediate feedback to the dev team.
- Review documentation: There should be peer and external reviews to ensure quality of documentation.
Testing as a function has undergone a paradigm shift and it is one of the disciplines that has a bright future as a career going forward. I believe strongly that exploratory testing where we use judgement and creative thinking to come up with high quality solutions has a strong future and one skill which will be difficult to automate.