Quality assurance consists of many activities, but often our primary goal is to verify and validate software our team is working on. Through various means we try to establish whether it fulfills customer requirements and other criteria like code quality, documentation, security, and so on. But who ensures that our job was done correctly? Even when we have a colleague who can review our test artifacts, they probably don't have enough time or context to do this thoroughly due to their own commitments.

Photo by Erik Mclean

This is why I always liked mutation testing as it aims to evaluate the quality of tests themselves. The idea is that we introduce small changes in the software under test and see if our tests managed to catch unexpected behavior by failing. We can, for instance, modify an SQL query from this:

select * from shopping_carts where user_id = ?

To this:

select * from shopping_carts where user_id != ?

So even though an insert method somewhere else would correctly add items to a shopping cart, it'll appear to be empty to the user due to the inverted where clause. If we don't have a test to validate the shopping cart after adding some items, all our tests will pass, revealing a gap in coverage. Like any other form of testing, mutation testing isn't exhaustive and it cannot prove the absence of bugs. Still, it can be a useful tool.

A couple of years ago, I created a simple website designed to assist QA engineers in practicing some of the most commonly used test design techniques: Equivalence Partitioning and Boundary Value Analysis. The site provides a specification of a feature along with multiple implementations; one is correct, and the others are not. The objective was to show that the correct test input can effectively detect all errors. However, my initial intent was not clear to the end users because I did not provide correct answers or any explanations! Let me do that in this post.

A few weeks ago, a potential employer asked me to complete a test assignment. The task was to implement a bunch of API and UI tests. This requirement is quite standard in the field, but I had not previously compiled any work ready to be shared alongside my CV.

Therefore, I made the decision not only to complete the assginment but also to document and explain it here in this blog post.