How to do System Testing
Testing the software system or software application as a whole is referred to as System Testing of the software. System testing of the application is done on complete application software to evaluate software's overall compliance with the business / functional / end-user requirements. The system testing comes under black box software testing. So, the knowledge of internal design or structure or code is not required for this type of software testing.
In system testing a software test professional aims to detect defects or bugs both within the interfaces and also within the software as a whole. However, the during integration testing of the application or software, the software test professional aims to detect the bugs / defects between the individual units that are integrated together.
During system testing, the focus is on the software design, behavior and even the believed expectations of the customer. So, we can also refer the system testing phase of software testing as investigatory testing phase of the software development life cycle.
At what stage of SDLC the System Testing comes into picture:
After the integration of all components of the software being developed, the whole software system is rigorously tested to ensure that it meets the specified business, functional & non-functional requirements. System Testing is build on the unit testing and integration testing levels. Generally, a separate and dedicated team is responsible for system testing. And, system testing is performed on stagging server.
Why system testing is required:
It is the first level of software testing where the software / application is tested as a whole.
It is done to verify and validate the technical, business, functional and non-functional requirements of the software. It also includes the verification & validation of software application architecture.
System testing is done on stagging environment that closely resembles the production environment where the final software will be deployed.
Entry Criteria for System Testing:
Unit Testing must be completed
Integration Testing must be completed
Complete software system should be developed
A software testing environment that closely resembling the production environment must be available (stagging environment).
System Testing in seven steps:
1.Creation of System Test Plan
2.Creation of system test cases
3.Selection / creation of test data for system testing
4.Software Test Automation of execution of automated test cases (if required)
5.Execution of test cases
6.Bug fixing and regression testing
7.Repeat the software test cycle (if required on multiple environments)
Contents of a system test plan: The contents of a software system test plan may vary from organization to organization or project to project. It depends how we have created the software test strategy, project plan and master test plan of the project. However, the basic contents of a software system test plan should be:
- Scope
- Goals & Objective
- Area of focus (Critical areas)
- Deliverables
- System testing strategy
- Schedule
- Entry and exit criteria
- Suspension & resumption criteria for software testing
- Test Environment
- Assumptions
- Staffing and Training Plan
- Roles and Responsibilities
- Glossary
How to write system test cases: The system test cases are written in a similar way as we write functional test cases. However, while creating system test cases following two points needs to be kept in mind:
- System test cases must cover the use cases and scenarios
- They must validate the all types of requirements - technical, UI, functional, non-functional, performance etc.
As per Wikipedia, there are total of 24 types of testings that needs to be considered during system testing. These are:
GUI software testing, Usability testing, Performance testing, Compatibility testing, Error handling testing, Load testing, Volume testing, Stress testing, User help testing, Security testing, Scalability testing, Capacity testing, Sanity testing, Smoke testing, Exploratory testing, Ad hoc testing, Regression testing, Reliability testing, Recovery testing, Installation testing, Idem potency testing, Maintenance testing, Recovery testing and failover testing, Accessibility testing
The format of system test cases contains:
Test Case ID - a unique number
Test Suite Name
Tester - name of tester who execute of write test cases
Requirement - Requirement Id or brief description of the functionality / requirement
How to Test - Steps to follow for execution of the test case
Test Data - Input Data
Expected Result
Actual Result
Pass / Fail
Test Iteration
Friday, March 26, 2010
Thursday, March 25, 2010
Testing School---Features that to be tested in OS Commerce website
Features that to be tested in OS Commerce website
Although osCommerce is still in its development stage, the available Milestone releases are considered to be stable with the following features:
General Functionality
Compatible with all PHP 4 versions
All features enabled by default for a complete out-of-the-box solution
Object oriented backend (3.0)
Completely multilingual with English, German, and Spanish provided by default
Setup / Installation
Automatic web-browser based installation and upgrade procedure
Design / Layout
Template struture implementation to:
allow layout changes to be adaptive, easy, and quickly to make (3.0)
allow easy integration into an existing site (3.0)
Support for dynamic images
Administration / Backend Functionality
Supports unlimited products and categories
Products-to-categories structure
Categories-to-categories structure
Add/Edit/Remove categories, products, manufacturers, customers, and reviews
Support for physical (shippable) and virtual (downloadable) products
Administration area secured with a username and password defined during installation
Contact customers directly via email or newsletters
Easily backup and restore the database
Print invoices and packaging lists from the order screen
Statistics for products and customers
Multilingual support
Multicurrency support
Automatically update currency exchange rates
Select what to display, and in what order, in the product listing page
Support for static and dynamic banners with full statistics
Customer / Frontend Functionality
All orders stored in the database for fast and efficient retrieval
Customers can view their order history and order statuses
Customers can maintain their accounts
Addressbook for multiple shipping and billing addresses
Temporary shopping cart for guests and permanent shopping cart for customers
Fast and friendly quick search and advanced search features
Product reviews for an interactive shopping experience
Forseen checkout procedure
Secure transactions with SSL
Number of products in each category can be shown or hidden
Global and per-category bestseller lists
Display what other customers have ordered with the current product shown
Breadcrumb trail for easy site navigation
Product Functionality
Dynamic product attributes relationship
HTML based product descriptions
Automated display of specials
Control if out of stock products can still be shown and are available for purchase
Customers can subscribe to products to receive related emails/newsletters
Payment Functionality
Accept numerous offline payment processing (cheque, money orders, offline credit care processing, ..)
Accept numerous online payment processing (PayPal, 2CheckOut, Authorize.net, iPayment, ..)
Disable certain payment services based on a zone basis
Shipping Functionality
Weight, price, and destination based shipping modules
Real-time quotes available (UPS, USPS, FedEx, ..)
Free shipping based on amount and destination
Disable certain shipping services based on a zone basis
Tax Functionality
Flexible tax implementation on a state and country basis
Set different tax rates for different products
Charge tax on shipping on a per shipping service basis
Although osCommerce is still in its development stage, the available Milestone releases are considered to be stable with the following features:
General Functionality
Compatible with all PHP 4 versions
All features enabled by default for a complete out-of-the-box solution
Object oriented backend (3.0)
Completely multilingual with English, German, and Spanish provided by default
Setup / Installation
Automatic web-browser based installation and upgrade procedure
Design / Layout
Template struture implementation to:
allow layout changes to be adaptive, easy, and quickly to make (3.0)
allow easy integration into an existing site (3.0)
Support for dynamic images
Administration / Backend Functionality
Supports unlimited products and categories
Products-to-categories structure
Categories-to-categories structure
Add/Edit/Remove categories, products, manufacturers, customers, and reviews
Support for physical (shippable) and virtual (downloadable) products
Administration area secured with a username and password defined during installation
Contact customers directly via email or newsletters
Easily backup and restore the database
Print invoices and packaging lists from the order screen
Statistics for products and customers
Multilingual support
Multicurrency support
Automatically update currency exchange rates
Select what to display, and in what order, in the product listing page
Support for static and dynamic banners with full statistics
Customer / Frontend Functionality
All orders stored in the database for fast and efficient retrieval
Customers can view their order history and order statuses
Customers can maintain their accounts
Addressbook for multiple shipping and billing addresses
Temporary shopping cart for guests and permanent shopping cart for customers
Fast and friendly quick search and advanced search features
Product reviews for an interactive shopping experience
Forseen checkout procedure
Secure transactions with SSL
Number of products in each category can be shown or hidden
Global and per-category bestseller lists
Display what other customers have ordered with the current product shown
Breadcrumb trail for easy site navigation
Product Functionality
Dynamic product attributes relationship
HTML based product descriptions
Automated display of specials
Control if out of stock products can still be shown and are available for purchase
Customers can subscribe to products to receive related emails/newsletters
Payment Functionality
Accept numerous offline payment processing (cheque, money orders, offline credit care processing, ..)
Accept numerous online payment processing (PayPal, 2CheckOut, Authorize.net, iPayment, ..)
Disable certain payment services based on a zone basis
Shipping Functionality
Weight, price, and destination based shipping modules
Real-time quotes available (UPS, USPS, FedEx, ..)
Free shipping based on amount and destination
Disable certain shipping services based on a zone basis
Tax Functionality
Flexible tax implementation on a state and country basis
Set different tax rates for different products
Charge tax on shipping on a per shipping service basis
Friday, March 5, 2010
Testing School---The complexity of testing
The complexity of testing
Using the following example we can explain why testing is so complicated.
You have a screen that contains one field only, named "username". The properties of the field are:
The field can contain only 6 alpha numeric characters and you have a keyboard with only alpha numeric keys.
The field is case sensitive.
Q: How many tests can you make in order to say that the screen is 100% bug free? Take a minute or two to answer this to yourself before you continue reading.
Well, the answer will be 56,800,235,586 (56 billion tests). Now, lets try to explain the answer:
The field contains:
6 places
The keyboard contains:
26 lower case chars
26 upper case chars
10 digits
We have 62 options for each place (the sum of keyboard options: 26 lower case chars + 26 upper case chars + 10 digits). But we have 6 places in the field so for each field we have 62 options and that means that the number of tests is:
62*62*62*62*62*62=62^6 = 56,800,235,584
Yes, this is the correct number of tests we need to perform in order to have 100% bugs free.
But we need to add 2 more tests:
one test is not to insert any value to the field.
the second test is to insert 7 chars to the field.
So the total number of tests is: 56,800,235,586.
If we will do 500 tests a day it will take us 311,234 years (if we will not jump from the window before).
Using the following example we can explain why testing is so complicated.
You have a screen that contains one field only, named "username". The properties of the field are:
The field can contain only 6 alpha numeric characters and you have a keyboard with only alpha numeric keys.
The field is case sensitive.
Q: How many tests can you make in order to say that the screen is 100% bug free? Take a minute or two to answer this to yourself before you continue reading.
Well, the answer will be 56,800,235,586 (56 billion tests). Now, lets try to explain the answer:
The field contains:
6 places
The keyboard contains:
26 lower case chars
26 upper case chars
10 digits
We have 62 options for each place (the sum of keyboard options: 26 lower case chars + 26 upper case chars + 10 digits). But we have 6 places in the field so for each field we have 62 options and that means that the number of tests is:
62*62*62*62*62*62=62^6 = 56,800,235,584
Yes, this is the correct number of tests we need to perform in order to have 100% bugs free.
But we need to add 2 more tests:
one test is not to insert any value to the field.
the second test is to insert 7 chars to the field.
So the total number of tests is: 56,800,235,586.
If we will do 500 tests a day it will take us 311,234 years (if we will not jump from the window before).
Testing School---All About Bugs and its reporting with Example
What is a bug?
A bug (also named 'fault' or 'defect') is an unexpected behavior in the software. A bug can be something the software should do but it doesn't do it or doesn't do it in the right way. A bug can also be that the software does something it shouldn't do. Usually, users don't like bugs. Occurrence of bugs, especially critical ones can cause users to stop working with the software.
Why bugs occur?
There are several reasons that can cause a bug. Let us mention some of them:
Unfinished requirements
Requirements that are not detailed enough
Requirements with multiple meanings
Logic errors in the design documents
Code errors
Not enough sufficient testing
Misunderstanding of user needs
Lack of documentation
Bug priority and severity?
We prioritize the bugs because we can't fix everything and we need to decide what we should fix first and that means that we prioritize what should be tested first. One parameter for prioritizing a bug is the bug's severity.
We define severity as the impact on the customer/end user. There are 4 common levels of severity:
Critical – software crash, hang, loss data, corrupt data
High – causes serious problems to the user
Medium – causes problem but the user can work around it
Low – trivial issues like spelling mistakes
Of course, you can add more severity levels as long as you well define each severity.
As testers we can add suggestions. A suggestion is not a bug. It is a request or a change that we think will give some added value to the system. If the project management will decide to add it then it will become a new requirement that will be tested as other requirements.
Let's take for example an office word software and review the bugs severity. A critical severity bug will be if the system will crash every time we send a document to the printer. A high severity bug will be if "Save" and "Save As" functions will not work. We understand that this unworking feature has a high impact on the end user. A medium severity bug will be if the "Save" function does not work, but the "Save As" function works well. The user can't do "Save" but he can work around it using the "Save As" function. A low severity bug will be if instead of "File/Send To" it will be written: "File/Sent To".
How to report a bug?
Why is it so important to report a bug?
Reporting a bug helps the programmer to reproduce the bug and to solve it.
It helps us to retest and see if the bug was fixed and if the fix is correct because we know what the bug was and how to try to reproduce it.
This is our product. We know how to find bugs. This is how we can show our knowledge and value to the project management.
You need to report a bug in a certain way. What can happen if we will report the bug in an improper way?
The programmer will not be able to reproduce it and fix it.
We will not be able to test it or our testers coworkers will not be able to retest it again because they will not understand and we will not remember what we meant 3 month ago when we report it.
The programmers might think we are not professional testers because they can't reproduce the bug.
The ping pong game – a programmer deliver to the tester because he doesn't understand the bug, and the tester deliver back to the programmer until the bug is fixed or it is thrown away.
Steps that are needed to be done before you report a bug in the bug tracker system:
Try to reproduce the bug at least one more time.
Search the bug in the bug tracking software. If it exists, check if you can add valuable information, if it doesn't exist, you need to report it as a new bug.
Steps to write a bug:
Write a short and meaningful title which describes the bug.
Write a description of the bug, what should be the correct result, the error message and any information that can help the programmer to isolate the bug.
Write how to reproduce the bug by breaking steps logically:
Straightforward path to trigger the bug.
Describe the steps.
Check that the bug is reproducible
Very bad example:
I can’t login to the bank account. Try to go to the account and you will not be able to do so.
Good example:
I am unable to log in to the web site.
Open IE 6.5 on window XP.
Enter the bank account URL (www.bank.co.il) in the location bar and hit enter.
Insert "user1" in login name field.
Insert "123456" in password field.
Click “login”.
Result: nothing happens, the system does nothing.
It should log you in because this username and password are exist in the database.
It was a very high level description of how to report a bug. Usually we will add more information such as:
Short but clear and explanatory short description – the reader should know the essence of the bug from it.
Enviroment setup - in here we will add the related configuration dimensions such as: site, setup, all software used and versions, all hardware used etc.
Full description of the bug - try not to repeat the short description of the bug. In here we will explain how the bug affects the end user, if it is not trivial, and what cannot the user do due to the bug. In here we will also explain if the bug has a traffic affecting, feature blocker, management affecting, test stopper etc. In addition we will need to write if we can work around the bug and if so, to explain in details how to work around it.
Full scenario for reproduction the steps made by the tester – attach the test case from the STD. Here you should also attach all links or relevant files, images, core files and logs.
Remember that many people need to read and understand your bug so you need to give any information that can help them to understand the bug, to reproduce it and to fix it.
Because many people read it, try to write it clearly and simple as you can and don't forget to use the speller (I usually write all bug info in word document, fix all typos and then copy it to the QC/TD/bug tracker system).
A bug (also named 'fault' or 'defect') is an unexpected behavior in the software. A bug can be something the software should do but it doesn't do it or doesn't do it in the right way. A bug can also be that the software does something it shouldn't do. Usually, users don't like bugs. Occurrence of bugs, especially critical ones can cause users to stop working with the software.
Why bugs occur?
There are several reasons that can cause a bug. Let us mention some of them:
Unfinished requirements
Requirements that are not detailed enough
Requirements with multiple meanings
Logic errors in the design documents
Code errors
Not enough sufficient testing
Misunderstanding of user needs
Lack of documentation
Bug priority and severity?
We prioritize the bugs because we can't fix everything and we need to decide what we should fix first and that means that we prioritize what should be tested first. One parameter for prioritizing a bug is the bug's severity.
We define severity as the impact on the customer/end user. There are 4 common levels of severity:
Critical – software crash, hang, loss data, corrupt data
High – causes serious problems to the user
Medium – causes problem but the user can work around it
Low – trivial issues like spelling mistakes
Of course, you can add more severity levels as long as you well define each severity.
As testers we can add suggestions. A suggestion is not a bug. It is a request or a change that we think will give some added value to the system. If the project management will decide to add it then it will become a new requirement that will be tested as other requirements.
Let's take for example an office word software and review the bugs severity. A critical severity bug will be if the system will crash every time we send a document to the printer. A high severity bug will be if "Save" and "Save As" functions will not work. We understand that this unworking feature has a high impact on the end user. A medium severity bug will be if the "Save" function does not work, but the "Save As" function works well. The user can't do "Save" but he can work around it using the "Save As" function. A low severity bug will be if instead of "File/Send To" it will be written: "File/Sent To".
How to report a bug?
Why is it so important to report a bug?
Reporting a bug helps the programmer to reproduce the bug and to solve it.
It helps us to retest and see if the bug was fixed and if the fix is correct because we know what the bug was and how to try to reproduce it.
This is our product. We know how to find bugs. This is how we can show our knowledge and value to the project management.
You need to report a bug in a certain way. What can happen if we will report the bug in an improper way?
The programmer will not be able to reproduce it and fix it.
We will not be able to test it or our testers coworkers will not be able to retest it again because they will not understand and we will not remember what we meant 3 month ago when we report it.
The programmers might think we are not professional testers because they can't reproduce the bug.
The ping pong game – a programmer deliver to the tester because he doesn't understand the bug, and the tester deliver back to the programmer until the bug is fixed or it is thrown away.
Steps that are needed to be done before you report a bug in the bug tracker system:
Try to reproduce the bug at least one more time.
Search the bug in the bug tracking software. If it exists, check if you can add valuable information, if it doesn't exist, you need to report it as a new bug.
Steps to write a bug:
Write a short and meaningful title which describes the bug.
Write a description of the bug, what should be the correct result, the error message and any information that can help the programmer to isolate the bug.
Write how to reproduce the bug by breaking steps logically:
Straightforward path to trigger the bug.
Describe the steps.
Check that the bug is reproducible
Very bad example:
I can’t login to the bank account. Try to go to the account and you will not be able to do so.
Good example:
I am unable to log in to the web site.
Open IE 6.5 on window XP.
Enter the bank account URL (www.bank.co.il) in the location bar and hit enter.
Insert "user1" in login name field.
Insert "123456" in password field.
Click “login”.
Result: nothing happens, the system does nothing.
It should log you in because this username and password are exist in the database.
It was a very high level description of how to report a bug. Usually we will add more information such as:
Short but clear and explanatory short description – the reader should know the essence of the bug from it.
Enviroment setup - in here we will add the related configuration dimensions such as: site, setup, all software used and versions, all hardware used etc.
Full description of the bug - try not to repeat the short description of the bug. In here we will explain how the bug affects the end user, if it is not trivial, and what cannot the user do due to the bug. In here we will also explain if the bug has a traffic affecting, feature blocker, management affecting, test stopper etc. In addition we will need to write if we can work around the bug and if so, to explain in details how to work around it.
Full scenario for reproduction the steps made by the tester – attach the test case from the STD. Here you should also attach all links or relevant files, images, core files and logs.
Remember that many people need to read and understand your bug so you need to give any information that can help them to understand the bug, to reproduce it and to fix it.
Because many people read it, try to write it clearly and simple as you can and don't forget to use the speller (I usually write all bug info in word document, fix all typos and then copy it to the QC/TD/bug tracker system).
Testing School---Boundary Values Testing with example
What is “Boundary Values Testing”?
“Boundary Values Testing” is a method that tests the boundary whether it's an input, an output or a performance boundary. Our tests focus on the boundaries values and not on the entire range of data. We will use it when we have a field that can contain a range of values as an input, an output or as a requirement.
How to use “Boundary Values Testing”?
If you have a range: (a to b), you will test the following:
Test Case
Value
Expected Result
1
a-1
Invalid
2
a
Valid
3
a+1
Valid
4
b-1
Valid
5
b
Valid
6
b+1
Invalid
According to the ISTQB in "Boundary Values Testing" we only test the following:
Test Case
Value
Expected Result
1
a-1
Invalid
2
a
Valid
3
b
Valid
4
b+1
Invalid
Why ISTQB has less test cases, because we can say that if 'a' and 'a-1' are working well, then we can assume that 'a+1' is working well. The same claim will be about the upper bound. If 'b' and 'b+1' are working well, then we can assume that 'b-1' is working well.
Any option of implementing the "Boundary Values" method you will use is good. Instead of testing the entire range, you can tests 6 or 4 cases and still have confiedence that the software works well.
Here are some more rules from my experience that you can take in consideration:
Always test 0 if it is inside the range and sometimes even if it's out of range because 0 has special effect on software (like dividing in 0).
Always test the empty string if it is inside the range and sometimes even if it's out of range.
Sometimes you can test a value that exists inside the range and not in the boundary just in case…(It allows you to sleep deeper at night…).
Now, let us practice the "Boundary Values Testing" method.
Practice 1
You are testing inventory software that contains a field of the quantity of items. The field can contain any value between 10 to 100 units.
What is the max number of test cases you will need in order to test the field?
What is the minimum number of test cases you will need in order to test the field, using boundary testing?
Try to solve it and then continue to read the answer.
Practice 1 - answer
The field contains a range of 10 to 100 (10-100).
The max number of test cases you will need in order to test the field will be 93 test cases (9, 10, 11, 12…97, 98, 99, 100, 10) or 94 test cases if we include the value of 0.
The minimum number of test cases you will need in order to test the field, using boundary testing will be 6 test cases (9, 10, 11, 99, 100, 101) or 7 test cases if we include 0.
Note that we need to add more tests like alphabetic chars and special chars like %, *.
Practice 2
You have a password field that can contain up to 8 characters and must have at least 3 characters. What is the minimum number of test cases you will need in order to test the field? (Pay attention to the requirement that specifies the field's length and not what kind of chars it can get! In the real world we can't ignore it but in order to simplify the example we will ignore it).
Try to solve it and then continue to read the answer.
Practice 2 - answer
6 test cases: length 2, length 3, length 4, length 7, length 8, length 9 or 7 test cases Length 2, length 3, length 4, length 6, length 7, length 8, length 9. We can add a test case that test the empty string.
Practice 3
You have a field that can contain up to 5 digits and can have 0 digits. The value of the field can be in the range of (-5148 to +3544) What is the minimum number of test cases you will need in order to test the field using "Boundary" testing?
Try to solve it and then continue to read the answer.
Practice 3 - answer
For the length we have 5 test cases: 0 length, 1 length, 4 length, 5 length and 6 length.
For the range we have 7 test cases: -5149, -5148, -5147, 0, +3543, +3544, +3545.
Total of 12 tests cases.
we reduce the amount of test cases to be less then 12 (try to solve it and continue to read)?
Well, we can reduce it to 10 test cases by combine the value testing of the field with the length testing of the field.
Case # Length Value Expected Result
1 0 None Valid
2 1 0 Valid
3 4 3543 Valid
4 4 3544 Valid
5 4 3545 Invalid
6 5 -5149 Invalid
7 5 -5148 Valid
8 5 -5147 Valid
9 6 123456 Invalid
10 6 -45322 Invalid
This was an example of why you must use your head all the time, because sometimes you can combine methodologies with each other or combine them with your common sense and reduce the amount of testing or create smart tests that will reveal beautiful and important bugs.
This example will work only if you know a little about how the programmer implemented the code (Gray box testing). If the programmer validate the length and the value together, then the reduction to 10 test cases is good.
“Boundary Values Testing” is a method that tests the boundary whether it's an input, an output or a performance boundary. Our tests focus on the boundaries values and not on the entire range of data. We will use it when we have a field that can contain a range of values as an input, an output or as a requirement.
How to use “Boundary Values Testing”?
If you have a range: (a to b), you will test the following:
Test Case
Value
Expected Result
1
a-1
Invalid
2
a
Valid
3
a+1
Valid
4
b-1
Valid
5
b
Valid
6
b+1
Invalid
According to the ISTQB in "Boundary Values Testing" we only test the following:
Test Case
Value
Expected Result
1
a-1
Invalid
2
a
Valid
3
b
Valid
4
b+1
Invalid
Why ISTQB has less test cases, because we can say that if 'a' and 'a-1' are working well, then we can assume that 'a+1' is working well. The same claim will be about the upper bound. If 'b' and 'b+1' are working well, then we can assume that 'b-1' is working well.
Any option of implementing the "Boundary Values" method you will use is good. Instead of testing the entire range, you can tests 6 or 4 cases and still have confiedence that the software works well.
Here are some more rules from my experience that you can take in consideration:
Always test 0 if it is inside the range and sometimes even if it's out of range because 0 has special effect on software (like dividing in 0).
Always test the empty string if it is inside the range and sometimes even if it's out of range.
Sometimes you can test a value that exists inside the range and not in the boundary just in case…(It allows you to sleep deeper at night…).
Now, let us practice the "Boundary Values Testing" method.
Practice 1
You are testing inventory software that contains a field of the quantity of items. The field can contain any value between 10 to 100 units.
What is the max number of test cases you will need in order to test the field?
What is the minimum number of test cases you will need in order to test the field, using boundary testing?
Try to solve it and then continue to read the answer.
Practice 1 - answer
The field contains a range of 10 to 100 (10-100).
The max number of test cases you will need in order to test the field will be 93 test cases (9, 10, 11, 12…97, 98, 99, 100, 10) or 94 test cases if we include the value of 0.
The minimum number of test cases you will need in order to test the field, using boundary testing will be 6 test cases (9, 10, 11, 99, 100, 101) or 7 test cases if we include 0.
Note that we need to add more tests like alphabetic chars and special chars like %, *.
Practice 2
You have a password field that can contain up to 8 characters and must have at least 3 characters. What is the minimum number of test cases you will need in order to test the field? (Pay attention to the requirement that specifies the field's length and not what kind of chars it can get! In the real world we can't ignore it but in order to simplify the example we will ignore it).
Try to solve it and then continue to read the answer.
Practice 2 - answer
6 test cases: length 2, length 3, length 4, length 7, length 8, length 9 or 7 test cases Length 2, length 3, length 4, length 6, length 7, length 8, length 9. We can add a test case that test the empty string.
Practice 3
You have a field that can contain up to 5 digits and can have 0 digits. The value of the field can be in the range of (-5148 to +3544) What is the minimum number of test cases you will need in order to test the field using "Boundary" testing?
Try to solve it and then continue to read the answer.
Practice 3 - answer
For the length we have 5 test cases: 0 length, 1 length, 4 length, 5 length and 6 length.
For the range we have 7 test cases: -5149, -5148, -5147, 0, +3543, +3544, +3545.
Total of 12 tests cases.
we reduce the amount of test cases to be less then 12 (try to solve it and continue to read)?
Well, we can reduce it to 10 test cases by combine the value testing of the field with the length testing of the field.
Case # Length Value Expected Result
1 0 None Valid
2 1 0 Valid
3 4 3543 Valid
4 4 3544 Valid
5 4 3545 Invalid
6 5 -5149 Invalid
7 5 -5148 Valid
8 5 -5147 Valid
9 6 123456 Invalid
10 6 -45322 Invalid
This was an example of why you must use your head all the time, because sometimes you can combine methodologies with each other or combine them with your common sense and reduce the amount of testing or create smart tests that will reveal beautiful and important bugs.
This example will work only if you know a little about how the programmer implemented the code (Gray box testing). If the programmer validate the length and the value together, then the reduction to 10 test cases is good.
Testing School---Equivalence Class Partitioning with example
What is "Equivalence Class Partitioning"?
We define "Equivalence Class Partitioning" as a method that can help you derive test cases. You identify classes of input or output conditions. The rule is that each member in the class causes the same kind of behavior of the system. In other words, the "Equivalence Class Partitioning" method creates sets of inputs or outputs that are handled in the same way by the application.
Another definition taken from Wikipedia:
"A technique in black box testing. It is designed to minimize the number of test cases by dividing tests in such a way that the system is expected to act the same way for all tests of each equivalence partition. Test inputs are selected from each class. Every possible input belongs to one and only one equivalence partition."
Why learn "Equivalence Class Partitioning"?
This method drastically reduces the number of test cases that are required to be tested because we don't have time, money or manpower to test everything. In addition, it can help you find many errors with the smallest number of test cases.
How to use "Equivalence Class Partitioning"?
There are 2 major steps we need to do in order to use equivalence class partitioning:
Identify the equivalence classes of input or output. Take each input's or output's condition that is described in the specification and derive at least 2 classes for it:
One class that satisfies the condition – the valid class.
Second class that doesn't satisfy the condition – the invalid class.
Design test cases based on the equivalence classes.
Example 1
In a computer store, the computer item can have a quantity between -500 to +500. What are the equivalence classes?
Answer: Valid class: -500 <= QTY <= +500
Invalid class: QTY > +500
Invalid class: QTY < -500
Example 2
In a computer store, the computer item type can be P2, P3, P4, and P5 (each type influences the price). What are the equivalence classes?
Answer: Valid class: type is P2
Valid class: type is P3
Valid class: type is P4
Valid class: type is P5
Invalid class: type isn’t P2, P3, P4 or P5
Practice
Bank account can be 500 to 1000 or 0 to 499 or 2000 (the field type is integer). What are the equivalence classes?
Try to solve it before reading the answer.
Practice 1 - answer
Valid class: 0 <= account <= 499
Valid class: 500 <= account <= 1000
Valid class: 2000 <= account <= 2000
Invalid class: account < 0
Invalid class: 1000 < account < 2000
Invalid class: account > 2000
Equivalence Class Vs Boundary Testing
Let us discuss about the difference between Equivalence class and boundary testing. For the discussion we will use the practice question:
Bank account can be integer in the following ranges: 500 to 1000 or 0 to 499 or 2000. What are the equivalence classes?
Answer:
valid class: 0 <= account <= 499
valid class: 500 <= account <= 1000
valid class: 2000 <= account <= 2000
invalid class: account < 0
invalid class: 1000 < account < 2000
invalid class: account > 2000
In equivalence class, you need to take one value from each class and test whether the value causes the system to act as the class' definition. It means that in this example, you need to create at least 6 test cases – one for each valid class and one for each invalid class.
How many test cases will be, if you use boundary testing?
The following table shows how much test cases will be using "Boundary Testing" method:
Test Case #
Value
Result
1
-1 Invalid
2
0 Valid
3
1 Valid
4
498 Valid
5
499 Valid
6
500 Valid
7
501 Valid
8
999 Valid
9 1000 Valid
10 1001 Invalid
11 1999 Invalid
12 2000 Valid
13 2001 Invalid
In boundary testing, you need to test each value in the boundary and you know the value, you don't need to choose it from any set. In this example you have 13 test cases.
Now, let us exam how to combine this 2 methods together.
The following table shows all the boundary testing values and their equivalence classes:
#
Boundary Value
Equivalence Class
Result
1
-1 account < 0 Invalid
2
0 0 <= account <= 499 Valid
3
1 0 <= account <= 499 Valid
4
498 0 <= account <= 499 Valid
5
499 0 <= account <= 499 Valid
6
500 500 <= account <= 1000 Valid
7
501 500 <= account <= 1000 Valid
8
999 500 <= account <= 1000 Valid
9 1000 500 <= account <= 1000 Valid
10 1001 1000 < account < 2000 Invalid
11 1999 1000 < account < 2000 Invalid
12 2000 2000 <= account <= 2000 Valid
13 2001 account > 2000 Invalid
Now, we can reduce some of the test cases that belong to the same equivalence class. We can delete lines 3 and 4 which belong to equivalence class "0 <= account <= 499". We also can delete lines 7 and 8 hich belong to "500 <= account <= 1000". The new table will be:
# Boundary Value Equivalence Class Result
1 -1 account < 0 Invalid
2 0 0 <= account <= 499 Valid
5 499 0 <= account <= 499 Valid
6 500 500 <= account <= 1000 Valid
9 1000 500 <= account <= 1000 Valid
10 1001 1000 < account < 2000 Invalid
11 1999 1000 < account < 2000 Invalid
12 2000 2000 <= account <= 2000 Valid
13 2001 account > 2000 Invalid
You can even reduce more test cases although in my opinion, it is important to keep this table because it keeps a hard connection to the boundary testing. You can see in the table that I didn't reduce those test cases that are touch in the boundary itself of each range.
Let's reduce more test cases (just for the fun and for the practice (test case 5, 9 and 10):
# Boundary Value Equivalence Class Result
1 -1 account < 0 Invalid
2 0 0 <= account <= 499 Valid
6 500 500 <= account <= 1000 Valid
11 1999 1000 < account < 2000 Invalid
12 2000 2000 <= account <= 2000 Valid
13 2001 account > 2000 Invalid
Now, in this table, for each equivalence class, you choose one value that belongs to boundary testing.
A smart man once told me that when I write a test case and I using equivalence class partitioning, not to write specific values. Instead, he told me to write the classes and their expected results. By that, each time a tester will run the test case he will choose new candidates from each class. Using that working method we can promise that each running will contain new values.
We define "Equivalence Class Partitioning" as a method that can help you derive test cases. You identify classes of input or output conditions. The rule is that each member in the class causes the same kind of behavior of the system. In other words, the "Equivalence Class Partitioning" method creates sets of inputs or outputs that are handled in the same way by the application.
Another definition taken from Wikipedia:
"A technique in black box testing. It is designed to minimize the number of test cases by dividing tests in such a way that the system is expected to act the same way for all tests of each equivalence partition. Test inputs are selected from each class. Every possible input belongs to one and only one equivalence partition."
Why learn "Equivalence Class Partitioning"?
This method drastically reduces the number of test cases that are required to be tested because we don't have time, money or manpower to test everything. In addition, it can help you find many errors with the smallest number of test cases.
How to use "Equivalence Class Partitioning"?
There are 2 major steps we need to do in order to use equivalence class partitioning:
Identify the equivalence classes of input or output. Take each input's or output's condition that is described in the specification and derive at least 2 classes for it:
One class that satisfies the condition – the valid class.
Second class that doesn't satisfy the condition – the invalid class.
Design test cases based on the equivalence classes.
Example 1
In a computer store, the computer item can have a quantity between -500 to +500. What are the equivalence classes?
Answer: Valid class: -500 <= QTY <= +500
Invalid class: QTY > +500
Invalid class: QTY < -500
Example 2
In a computer store, the computer item type can be P2, P3, P4, and P5 (each type influences the price). What are the equivalence classes?
Answer: Valid class: type is P2
Valid class: type is P3
Valid class: type is P4
Valid class: type is P5
Invalid class: type isn’t P2, P3, P4 or P5
Practice
Bank account can be 500 to 1000 or 0 to 499 or 2000 (the field type is integer). What are the equivalence classes?
Try to solve it before reading the answer.
Practice 1 - answer
Valid class: 0 <= account <= 499
Valid class: 500 <= account <= 1000
Valid class: 2000 <= account <= 2000
Invalid class: account < 0
Invalid class: 1000 < account < 2000
Invalid class: account > 2000
Equivalence Class Vs Boundary Testing
Let us discuss about the difference between Equivalence class and boundary testing. For the discussion we will use the practice question:
Bank account can be integer in the following ranges: 500 to 1000 or 0 to 499 or 2000. What are the equivalence classes?
Answer:
valid class: 0 <= account <= 499
valid class: 500 <= account <= 1000
valid class: 2000 <= account <= 2000
invalid class: account < 0
invalid class: 1000 < account < 2000
invalid class: account > 2000
In equivalence class, you need to take one value from each class and test whether the value causes the system to act as the class' definition. It means that in this example, you need to create at least 6 test cases – one for each valid class and one for each invalid class.
How many test cases will be, if you use boundary testing?
The following table shows how much test cases will be using "Boundary Testing" method:
Test Case #
Value
Result
1
-1 Invalid
2
0 Valid
3
1 Valid
4
498 Valid
5
499 Valid
6
500 Valid
7
501 Valid
8
999 Valid
9 1000 Valid
10 1001 Invalid
11 1999 Invalid
12 2000 Valid
13 2001 Invalid
In boundary testing, you need to test each value in the boundary and you know the value, you don't need to choose it from any set. In this example you have 13 test cases.
Now, let us exam how to combine this 2 methods together.
The following table shows all the boundary testing values and their equivalence classes:
#
Boundary Value
Equivalence Class
Result
1
-1 account < 0 Invalid
2
0 0 <= account <= 499 Valid
3
1 0 <= account <= 499 Valid
4
498 0 <= account <= 499 Valid
5
499 0 <= account <= 499 Valid
6
500 500 <= account <= 1000 Valid
7
501 500 <= account <= 1000 Valid
8
999 500 <= account <= 1000 Valid
9 1000 500 <= account <= 1000 Valid
10 1001 1000 < account < 2000 Invalid
11 1999 1000 < account < 2000 Invalid
12 2000 2000 <= account <= 2000 Valid
13 2001 account > 2000 Invalid
Now, we can reduce some of the test cases that belong to the same equivalence class. We can delete lines 3 and 4 which belong to equivalence class "0 <= account <= 499". We also can delete lines 7 and 8 hich belong to "500 <= account <= 1000". The new table will be:
# Boundary Value Equivalence Class Result
1 -1 account < 0 Invalid
2 0 0 <= account <= 499 Valid
5 499 0 <= account <= 499 Valid
6 500 500 <= account <= 1000 Valid
9 1000 500 <= account <= 1000 Valid
10 1001 1000 < account < 2000 Invalid
11 1999 1000 < account < 2000 Invalid
12 2000 2000 <= account <= 2000 Valid
13 2001 account > 2000 Invalid
You can even reduce more test cases although in my opinion, it is important to keep this table because it keeps a hard connection to the boundary testing. You can see in the table that I didn't reduce those test cases that are touch in the boundary itself of each range.
Let's reduce more test cases (just for the fun and for the practice (test case 5, 9 and 10):
# Boundary Value Equivalence Class Result
1 -1 account < 0 Invalid
2 0 0 <= account <= 499 Valid
6 500 500 <= account <= 1000 Valid
11 1999 1000 < account < 2000 Invalid
12 2000 2000 <= account <= 2000 Valid
13 2001 account > 2000 Invalid
Now, in this table, for each equivalence class, you choose one value that belongs to boundary testing.
A smart man once told me that when I write a test case and I using equivalence class partitioning, not to write specific values. Instead, he told me to write the classes and their expected results. By that, each time a tester will run the test case he will choose new candidates from each class. Using that working method we can promise that each running will contain new values.
Testing School---All About Test Matrix....
What is a Test Matrix?
Matrices provide an easy structure for testing common issues. A common issue is an issue that repeats itself from project to project. Testing a common issue should be relevant to the project itself.
Examples for common issues are:
Fields (integer, dates, time, etc)
File names
Printing
Saving a file
Deleting a file
Sending a file
Login process
UI issues
Other
Why is it important to learn "Test Matrix"?
There are several reasons why to learn and to do test matrices.
It can reduce working time - once you have a matrix for a specific issue, it will take you less time to test it or to think how to test it.
It is logical and testing challenging - It is a challenge to make your own matrix and to find common issues that are repeatable from project to project.
In future projects it will save you time and it can give your more time to handle more complex issues.
It’s fun (for those who have a testing mania like the writer).
How to Do a Test Matrix?
The algorithm for creating a test matrix is:
Find an issue that repeats itself from project to project
Think of tests that you routinely perform on this issue
Sort all the tests and put them in a matrix
Example
Let’s build a matrix for integer field. First, we will think and write all the tests that we can perform on an integer field:
0
Valid value
Lower boundary – 1
Lower boundary
Lower boundary + 1
Upper boundary – 1
Upper boundary
Upper boundary + 1
Nothing
Negative value
Special chars: < > ? , . / ; : ‘ “ [ ] { } \ | + = _ - ( ) * & ^ % $ # @ ! ~ `
Uppercase chars
Lowercase chars
Spaces
Leading spaces before the value
Value follows with spaces
Length lower boundary – 1
Length lower boundary
Length lower boundary + 1
Length upper boundary – 1
Length upper boundary
Length upper boundary + 1
Mix of digits, chars and spaces
…
Now we will insert them into a generic matrix:
Integer field matrix
Field Name
Cases
0
Valid value
Lower boundary – 1
Lower boundary
Lower boundary + 1
Upper boundary – 1
Upper boundary
Upper boundary + 1
Nothing
Negative value
Special chars: < > ? , . / ; : ‘ “ [ ] { } \ | + = _ - ( ) * & ^ % $ # @ ! ~ `
Uppercase chars
Lowercase chars
Spaces
Leading spaces before the value
Value follows with spaces
Length lower boundary – 1
Length lower boundary
Length lower boundary + 1
Length upper boundary – 1
Length upper boundary
Length upper boundary + 1
Mixed of digits, chars and spaces
…
Now let use it for a project that contains 2 integer fields: “Age”, “Price”. For each field we will mark those cases we want to test.
Integer field matrix
Field Name Age Price
Cases
0 X X
Valid value X
Lower boundary – 1 X
Lower boundary X
Lower boundary + 1
Upper boundary – 1
Upper boundary X
Upper boundary + 1 X
Nothing
Negative value X
Special chars: < > ? , . / ; : ‘ “ [ ] { } \ | + = _ - ( ) * & ^ % $ # @ ! ~ ` X X
Uppercase chars
Lowercase chars
Spaces X
Leading spaces before the value
Value follows with spaces
Length lower boundary – 1 X
Length lower boundary X
Length lower boundary + 1
Length upper boundary – 1 X X
Length upper boundary X
Length upper boundary + 1
Mixed of digits, chars and spaces X
…
This method gives you the power to change in each test cycle the cases that need to be tested for those fields, without investing many efforts on changing the STD document.
Practice 1
Create a matrix for saving a file. Try to solve it before continue reading.
Practice 1 - answer
Save a new file
Save a file with an existing file name
Save in another format
Save a file to a full disk
Save a file to a write protected disk
Save a file to a remote disk
Save a large file and during the saving process print the file
…
Practice 2
Create a matrix for a date field. Try to solve it before continue reading.
Practice 2 - answer
Insert chars
Insert numbers
Insert day/month 0
Insert day 32
Insert month 13
Insert year 90 (means 1990, 2090 ???)
Insert other format of dates:
24/12/1978 and 12/24/1978
Insert 16/9/2006 and 16.9.2006
…
Practice 3
Create a matrix for login a system: username and password. Try to solve it before continue reading.
Practice 3 - answer
Correct username and wrong password
Wrong username and correct password
Wrong username and wrong password
Correct username and correct password
Correct username and password like ‘select 1’
Uppercase and lowercase
…
Practice 4
Create a matrix for a char field with length x. Try to solve it before continue reading.
Practice 4 - answer
Similar to the integer example
Practice 5
Create a matrix for deleting a file. Try to solve it before continue reading.
Practice 5 - answer
Delete a file
Delete a very large file
Delete an empty file
Delete an empty folder
Delete a folder with many files
Delete an open file
Delete a file while sending other files to the printer
…
Practice 6
Create a matrix for testing an email field. Try to solve it before continue reading.
Practice 6 - answer
Insert a valid mail: a.a@a.com, a@123.co.il
Insert an invalid mail format
Insert chars
Insert numbers
Insert a very long email
…
Practice 7
Create a matrix for testing a new screen that will be insert into the application. Try to solve it before continue reading.
Practice 7 - answer
Test typo in screen title, buttons, fields and tables.
Test that keyboard shortcuts work well when buttons are enable and lock when buttons are disabled.
Check sorting at least 3 times on each column.
If you have a search criteria, combine a search with several fields using a method named "all pairs".
Create a test where the search is bring you one row, 2 rows, more then 2 rows and 0 rows.
Create a test for clear the search (brings all rows).
Create a search where in string/text field you are trying to insert a word contains the sign ' - if the programmer didn't handle it and you are working with XML files or with databases, you will get an error.
If you can search by "starting with" or "contains with" test spaces and '.
Test that each button is working as expected.
Test the paging feature if you screen support in regular view and in searching results - sometimes there is a bug that paging in search result will bring the entire rows.
Ttest the "yes", "no", "cancel" options in message box.
Try to active/inactive a radio button and a check box field.
Test that mandatory fields are really mandatory and that the message about it is correct.
Test the close window button.
Matrices provide an easy structure for testing common issues. A common issue is an issue that repeats itself from project to project. Testing a common issue should be relevant to the project itself.
Examples for common issues are:
Fields (integer, dates, time, etc)
File names
Printing
Saving a file
Deleting a file
Sending a file
Login process
UI issues
Other
Why is it important to learn "Test Matrix"?
There are several reasons why to learn and to do test matrices.
It can reduce working time - once you have a matrix for a specific issue, it will take you less time to test it or to think how to test it.
It is logical and testing challenging - It is a challenge to make your own matrix and to find common issues that are repeatable from project to project.
In future projects it will save you time and it can give your more time to handle more complex issues.
It’s fun (for those who have a testing mania like the writer).
How to Do a Test Matrix?
The algorithm for creating a test matrix is:
Find an issue that repeats itself from project to project
Think of tests that you routinely perform on this issue
Sort all the tests and put them in a matrix
Example
Let’s build a matrix for integer field. First, we will think and write all the tests that we can perform on an integer field:
0
Valid value
Lower boundary – 1
Lower boundary
Lower boundary + 1
Upper boundary – 1
Upper boundary
Upper boundary + 1
Nothing
Negative value
Special chars: < > ? , . / ; : ‘ “ [ ] { } \ | + = _ - ( ) * & ^ % $ # @ ! ~ `
Uppercase chars
Lowercase chars
Spaces
Leading spaces before the value
Value follows with spaces
Length lower boundary – 1
Length lower boundary
Length lower boundary + 1
Length upper boundary – 1
Length upper boundary
Length upper boundary + 1
Mix of digits, chars and spaces
…
Now we will insert them into a generic matrix:
Integer field matrix
Field Name
Cases
0
Valid value
Lower boundary – 1
Lower boundary
Lower boundary + 1
Upper boundary – 1
Upper boundary
Upper boundary + 1
Nothing
Negative value
Special chars: < > ? , . / ; : ‘ “ [ ] { } \ | + = _ - ( ) * & ^ % $ # @ ! ~ `
Uppercase chars
Lowercase chars
Spaces
Leading spaces before the value
Value follows with spaces
Length lower boundary – 1
Length lower boundary
Length lower boundary + 1
Length upper boundary – 1
Length upper boundary
Length upper boundary + 1
Mixed of digits, chars and spaces
…
Now let use it for a project that contains 2 integer fields: “Age”, “Price”. For each field we will mark those cases we want to test.
Integer field matrix
Field Name Age Price
Cases
0 X X
Valid value X
Lower boundary – 1 X
Lower boundary X
Lower boundary + 1
Upper boundary – 1
Upper boundary X
Upper boundary + 1 X
Nothing
Negative value X
Special chars: < > ? , . / ; : ‘ “ [ ] { } \ | + = _ - ( ) * & ^ % $ # @ ! ~ ` X X
Uppercase chars
Lowercase chars
Spaces X
Leading spaces before the value
Value follows with spaces
Length lower boundary – 1 X
Length lower boundary X
Length lower boundary + 1
Length upper boundary – 1 X X
Length upper boundary X
Length upper boundary + 1
Mixed of digits, chars and spaces X
…
This method gives you the power to change in each test cycle the cases that need to be tested for those fields, without investing many efforts on changing the STD document.
Practice 1
Create a matrix for saving a file. Try to solve it before continue reading.
Practice 1 - answer
Save a new file
Save a file with an existing file name
Save in another format
Save a file to a full disk
Save a file to a write protected disk
Save a file to a remote disk
Save a large file and during the saving process print the file
…
Practice 2
Create a matrix for a date field. Try to solve it before continue reading.
Practice 2 - answer
Insert chars
Insert numbers
Insert day/month 0
Insert day 32
Insert month 13
Insert year 90 (means 1990, 2090 ???)
Insert other format of dates:
24/12/1978 and 12/24/1978
Insert 16/9/2006 and 16.9.2006
…
Practice 3
Create a matrix for login a system: username and password. Try to solve it before continue reading.
Practice 3 - answer
Correct username and wrong password
Wrong username and correct password
Wrong username and wrong password
Correct username and correct password
Correct username and password like ‘select 1’
Uppercase and lowercase
…
Practice 4
Create a matrix for a char field with length x. Try to solve it before continue reading.
Practice 4 - answer
Similar to the integer example
Practice 5
Create a matrix for deleting a file. Try to solve it before continue reading.
Practice 5 - answer
Delete a file
Delete a very large file
Delete an empty file
Delete an empty folder
Delete a folder with many files
Delete an open file
Delete a file while sending other files to the printer
…
Practice 6
Create a matrix for testing an email field. Try to solve it before continue reading.
Practice 6 - answer
Insert a valid mail: a.a@a.com, a@123.co.il
Insert an invalid mail format
Insert chars
Insert numbers
Insert a very long email
…
Practice 7
Create a matrix for testing a new screen that will be insert into the application. Try to solve it before continue reading.
Practice 7 - answer
Test typo in screen title, buttons, fields and tables.
Test that keyboard shortcuts work well when buttons are enable and lock when buttons are disabled.
Check sorting at least 3 times on each column.
If you have a search criteria, combine a search with several fields using a method named "all pairs".
Create a test where the search is bring you one row, 2 rows, more then 2 rows and 0 rows.
Create a test for clear the search (brings all rows).
Create a search where in string/text field you are trying to insert a word contains the sign ' - if the programmer didn't handle it and you are working with XML files or with databases, you will get an error.
If you can search by "starting with" or "contains with" test spaces and '.
Test that each button is working as expected.
Test the paging feature if you screen support in regular view and in searching results - sometimes there is a bug that paging in search result will bring the entire rows.
Ttest the "yes", "no", "cancel" options in message box.
Try to active/inactive a radio button and a check box field.
Test that mandatory fields are really mandatory and that the message about it is correct.
Test the close window button.
Subscribe to:
Comments (Atom)