When testing software (or any type of system), we can use various approaches. Black box testing is a group of techniques that should be part of any tester’s toolbox. But don’t worry if you’re not familiar with it yet. As you’ll see in this article, the basic techniques aren’t too difficult. And you might already be doing black box testing without knowing it!
A Definition
Before we go any further, it’s useful to define what we mean by black box testing:
In black box testing, we test a system without knowing or looking at how the system works internally.
What this means is that we know how the system should react to certain inputs, but we don’t know or care what happens inside.
Black Box Testing Types
Black box testing can be applied to many types of tests. Even tests that are mostly written by developers who know how the system works internally: unit and integration tests. But black box testing works great for testers who haven’t written the code and are unbiased when designing tests.
Let’s look at some types of tests that are ideal for black box testing.
Business Acceptance Tests
This is probably what many people think of when they talk about black box tests. Business acceptance tests (sometimes just called acceptance tests) are tests that are based on requirements and scenarios written by the business users.
They can be automated with tools like Testim or Cucumber. They often represent a user journey through the system. For example:
- log in
- go to the “add new customer “page”
- fill in certain details
- click save
- go to the customer overview page
- verify the customer is in the list
With Testim, you could record these steps as you walk through the web application. Once you have recorded the scenario, you can replay it, to verify that everything still works as expected.
When developers add new features or change existing pieces of code, these scenarios can act as a safety net. The tests won’t catch every possible regression bug, but they can give you some assurance that the end-user can still perform the critical actions that you’ve defined.
Performance Tests
Performance tests perform a series of tests to see how the application performs. They are less concerned with the specific outcome, but more with metrics like response times, memory or CPU usage. For this, the tester doesn’t need to know the internal details of the system.
With tools like JMeter, you can script performance and load tests. In load tests, you will put a heavy load on your system to see if it still responds in a timely manner. This is especially useful for web applications that should be able to handle many concurrent requests.
Black Box Penetration Testing
A penetration test, or pentest, is when someone tries to gain access to the system or tries to take it down. Of course, this person needs some knowledge of hacking and has the authorization to perform this test.
In a black box penetration test, the hacker is asked to hack the system without prior knowledge. They take on the same position as a malicious hacker. This is in contrast to white box penetration testing where the hacker can have full access to the source code.
Black Box Testing Techniques
Black box testing isn’t concerned with the source code or the database layout. This is why we can design these tests as soon as the requirements have been defined. Even before the developers start writing the code.
Let’s look at some techniques to design test scenarios.
Decision Table Testing
This is a technique that doesn’t require a lot of effort. All you need to do is create a table that has the possible values of your inputs and the expected output.
For example, if we have a business rule to determine the discount for a customer, we might make an overview like this:
Given our inputs (the member status and whether or not it’s the member’s birthday), we can define what the expected discount should be. A decision table provides us with an overview of the cases we should be testing.
Of course, the amount of combinations increases when you have more inputs and more possible values. That is where pairwise testing can help us.
Pairwise Testing
Pairwise testing is sometimes called all-pairs testing.
Most software bugs are caused by the combination of specific values of two parameters. It’s increasingly less common that bugs are caused by a combination of more parameters. This allows us to reduce the number of test cases significantly when many combinations of inputs are possible.
Let’s assume a system that accepts three parameters: a Boolean, one of three colors and a value between one and four. This gives us a total of 2 x 3 x 4 possible combinations, meaning we would have to write 24 test cases.
With pairwise testing, we can reduce this to 12 cases. The way to do so is to take the following steps:
- List the values of the parameter with the most possible values in a column. Repeat each value n times, where n is the number of possible values of the parameter with the second most possible values
- Then, in a second column, list the values of the second parameter and make sure you have made each possible combination with the first value.
In our example, we now have this table:
Now in a third column, add the possible values of the last parameter. Again, making sure to make each possible combination with the previous parameter. This gives us the resulting table with 12 test cases:
We went from 24 possible test cases to 12. The more parameters and possible values you have, the more you can gain from pairwise testing. To know how many test cases you should end up with, you can multiply the number of possible values of the two parameters with the most possible values.
You can read more about pairwise testing on pairwise.org. There is a tools section that contains links to some tools that can generate the test cases based on your inputs.
Equivalence Partitioning
Many applications have points of entry that accept a range of values. Let’s say a system has an entry point that accepts an integer between 0 and 10.
With these restrictions, we can identify three partitions:
- negative infinity to -1
- 0 to 10
- 11 to positive infinity
We can also say the inputs belong to a certain equivalence class. With these equivalence classes or partitions, you can define three test cases:
- a negative number, for example, -4,
- a valid number, for example, 5,
- an invalid positive number, for example, 12
Equivalence Partitioning doesn’t only apply to numbers. For example, an application that allows you to upload Word, Excel and PDF files has four partitions (including an invalid file type). If the actual content of the document doesn’t matter of course.
Boundary Values Analysis
Boundary Value Analysis builds on the Equivalence Partitioning technique. In Boundary Value Analysis we identify test cases at the boundaries of our partitions. The boundaries are where we transition from one partition to the other. Each boundary provides two test cases: one on each side of the boundary.
In our example, we would write test cases for
- -1
- 0
- 10
- 11
Boundary Value Analysis applies to ranges of values, less so to categories. In our file upload example above, we can’t really define boundaries for our file types.
State Transition Testing
If a system can only be in a limited amount of states and if it can move from one state to another based on some input and predefined rules, then it can be regarded as a “state machine.” Given this state machine, we can define a starting state, an input, and the expected resulting state. These scenarios will define our tests.
For example, we can describe a simple media player as a state machine. To keep it simple, this media player has only three commands: play, stop and pause. This results in the following state machine:
This diagram results in the following state transition table:
This final table now lists our test scenarios. Each test will set up the system in a given state, send the command to the system and verify the new state.
A state transition table is similar to a decision table. The difference is that in a decision table you can have multiple inputs and outputs per scenario, whereas in a state transition table you have one starting state, one action, and one final state per scenario.
Exploratory Testing
Exploratory testing is a mental exercise in which the tester takes time to think about new and useful test cases. The testers get a high level of autonomy and responsibility here. It’s their task to design new test scenarios based on prior experience and their knowledge about the system.
The tester can have knowledge about how the system worked in the past and how they expect the system to behave. They don’t need any knowledge about the code or infrastructure.
A simple example is when a tester designs scenario’s to check inputs that often generate errors: entering an invalid email address, double-clicking a save button to check if the input isn’t saved twice, etc. An experienced tester may come up with scenarios that aren’t covered by any of our previous techniques and may have been overlooked by the developers.
Combining Techniques
The test types and techniques above can be combined. For example, when you have inputs with both ranges and fixed values, you can use the pairwise technique with boundary value analysis.
Go ahead and apply these techniques and you’ll uncover many new test scenarios and improve the quality of your software!
This post was written by Peter Morlion. Peter is a passionate programmer that helps people and companies improve the quality of their code, especially in legacy codebases. He firmly believes that industry best practices are invaluable when working towards this goal, and his specialties include TDD, DI, and SOLID principles.
