Performance Bugs: How to find? How to report?
In this article we are going to know how to find bugs related to performance as well as how to report them. This is very basic checklist which will differ based on your project. I will try to give example with asp.net web application.
To know about basic bug and reporting , you can see one my previous posts which tells us all about a bug report.
Performance Bugs :
This is kind of admiring that Performance bugs are hard to find. Very difficult to get the evidence. So, if you are finding bugs, you have to be preferred with full performance goal and requirements.
And, you have to monitor your application as well as resources(CPU/Memory/Disk/IO etc).
While monitoring resources, you have to see the pattern of resource uses and application behavior.
You need to make heuristics analysis either with pen and paper or inside monitor tool to observe the change behaviors.
This way you can get the scenario or scope of area for possible bug. It's more like investigation and waiting for possible incident(like Sharlok Homes).
Before doing that, you must know the Architecture of your
->Application (Development and Deployment model, framework used, platform architecture, security modules etc)
->Infrastructure(How deployed, environment, which code/modules executes where & how)
And you also should know, why you are testing in every aspect (Goals and Requirement)
Example : I consider following for an individual request to find bug for my last project(web)
1. What is the size of the request?
2. How much time it needs for going to server and coming back with http 200 messages(server get the request) and with responses.
3. What are parameters we are sending as request in Http body, URL and Headers?
4. Why we are sending which parameters?
a. Categorizing those parameters,
i. Which parameters are based on client behavior/step
ii. Which parameters are based on client data?
iii. Which are based on environment
iv.Which are based on our application implementation
b. Finding unnecessary parameters which browser send from client and how can we eliminate them.(optimization)
i. Eliminating unnecessary parameters(based on business logic)
ii. Optimize necessary parameters(like view state, call backs)
c. How can we make those requests more precise and optimize for communication.
d. Organizing resources(JS/AJAX/Image/CSS/JQuery)
e. Compress the requests and responses
f. Using long time cookie for static resources with expire ending.
5. Why we are sending request in this manner? can we send in any other way to keep the application faster?(like replacing post back calls with Ajax)
How to get the Evidence?
Difficult bugs comes with much harder traceability...:)
So, as it is hard to find, you have prepare for evidence. Usually, performance bug evidence can be found in these aspects.
a. While debugging application from client and server for defined test steps. This should be done with tools as well as manually. (before performance test script generation in tool)
b. Monitoring resources and application while performance testing(like : IIS/apache, hosted PC, Application , DB for a web application)
c. Analyzing Reports generated by performance tools after test execution
d. Client side single user execution during high /peak load test is going on. You have to have debug tools enabled along with browser monitoring tools(web product)/proxy based tools.
So, to get evidence,
->I suggest to be prepared for Screenshots/Video recording for analysis later on.
->Make some Log analysis scripts or parsers to have quick log analysis.
->Be prepare with Pen & Paper to make quick notes.
->Make you mind for observing interesting important items which you see and related to goal
->Try not to interrupt test execution, better to let it finish
->Try long term test execution , when you need to test capacity of your application . I prefer 12/24 hour run over weekends.
->Try mixing up load scenario for different execution, More user, less time, more user long time, constant user increment, user high-low combination(like spike) , combining high load and security scanning etc
This is fully depend on application that you are working with. It may not be based on my experiences. Try to get more load scenarios involving with your application architecture & performance goals.
How to write a performance bug report? :
This is interesting, usually bugs are very specific to UI or user interactions. But, as performance testing is more related to application architecture testing. So, performance bugs should also have reference with application execution as well as how it happened from architectural prospective. That means, this standard procedure of bug reporting and + Something. And This "something" part should be developer friendly as well as full detail with references(as developers will read and get find what you are trying to explain). Let's see more detail on the "Something" with example.
Example , for a web application with http protocol, when you click a button, a function bug will refer to only the use case, but a performance bug should also refer to what request we send to server & received from server in details.
You should add reference on the requested URL, header details, body details. It should have detail explanation on those areas.
For this example, let say a http request Body has 25 parameters & 21 of them with values. So,
- What are those parameters,
- What are the reference of the parameters,
- Why those values are there.
Same justification for request header also. Up to this, it is only description.
So, what will be inside evidence? It should reflect your requirements. Let's say you need to know size, time and capacity for this request and you get the bug. So, in bug report you should provide
a. The total size of the request send to server
b. Time taken for server processing(http 200 response) for this requests
c. How the time varies in case of different load scenario
d. How many responses are received for this request.
e. Individual and total size of all of those responses.
f. Individual and total time of all of those responses for different load scenarios
g.What is the rendering time for those request(Time from individual single user perform the same task in browser – server response time calculated from server)
h. For Rendering, which response takes how much time in browser(from browser tool, or fiddler or proxy)
And, From individual request, you need to break down each parameter detail and possible cause for taking the amount of time that the request is taking.
For advance QAs : (Root Cause Analysis & Suggestions) :
You can go more further, for that you might need to have access of the code. You have to include profiler and should see what cause taking more time for that single request processing.
For web application, use profiler in IIS as well as in code base to pinpoint each request impact. Like, we are doing in our project , tracing from IIS, prove with profiler in IIS, tracing respective event inside code with code analysis tools and provide evidence on what cause the extra time.
Some, senior QAs can go more further. They can provide possible solution to overcome situation.
Example : In our project, we have found , we are not using viewstate(asp.net) properly that cases a huge time. So, we are in need for optimizing view state. But, there are several ways to do. So, we are suggesting different ways as well as impact of the ways for solving. This involve retesting ,that verifies the bug . This part should consider business requirements as well as security requirements.
So , summary, a bug should contains steps to reproduce and standard items mandatory for a function bug , as well as Screenshots, Logs, Charts, test results, Analysis reports, Code snipped, Demo projects, tool references etc.
Thanks...:)
To know about basic bug and reporting , you can see one my previous posts which tells us all about a bug report.
Performance Bugs :
This is kind of admiring that Performance bugs are hard to find. Very difficult to get the evidence. So, if you are finding bugs, you have to be preferred with full performance goal and requirements.
And, you have to monitor your application as well as resources(CPU/Memory/Disk/IO etc).
While monitoring resources, you have to see the pattern of resource uses and application behavior.
You need to make heuristics analysis either with pen and paper or inside monitor tool to observe the change behaviors.
This way you can get the scenario or scope of area for possible bug. It's more like investigation and waiting for possible incident(like Sharlok Homes).
Before doing that, you must know the Architecture of your
->Application (Development and Deployment model, framework used, platform architecture, security modules etc)
->Infrastructure(How deployed, environment, which code/modules executes where & how)
And you also should know, why you are testing in every aspect (Goals and Requirement)
Example : I consider following for an individual request to find bug for my last project(web)
1. What is the size of the request?
2. How much time it needs for going to server and coming back with http 200 messages(server get the request) and with responses.
3. What are parameters we are sending as request in Http body, URL and Headers?
4. Why we are sending which parameters?
a. Categorizing those parameters,
i. Which parameters are based on client behavior/step
ii. Which parameters are based on client data?
iii. Which are based on environment
iv.Which are based on our application implementation
b. Finding unnecessary parameters which browser send from client and how can we eliminate them.(optimization)
i. Eliminating unnecessary parameters(based on business logic)
ii. Optimize necessary parameters(like view state, call backs)
c. How can we make those requests more precise and optimize for communication.
d. Organizing resources(JS/AJAX/Image/CSS/JQuery)
e. Compress the requests and responses
f. Using long time cookie for static resources with expire ending.
5. Why we are sending request in this manner? can we send in any other way to keep the application faster?(like replacing post back calls with Ajax)
How to get the Evidence?
Difficult bugs comes with much harder traceability...:)
So, as it is hard to find, you have prepare for evidence. Usually, performance bug evidence can be found in these aspects.
a. While debugging application from client and server for defined test steps. This should be done with tools as well as manually. (before performance test script generation in tool)
b. Monitoring resources and application while performance testing(like : IIS/apache, hosted PC, Application , DB for a web application)
c. Analyzing Reports generated by performance tools after test execution
d. Client side single user execution during high /peak load test is going on. You have to have debug tools enabled along with browser monitoring tools(web product)/proxy based tools.
So, to get evidence,
->I suggest to be prepared for Screenshots/Video recording for analysis later on.
->Make some Log analysis scripts or parsers to have quick log analysis.
->Be prepare with Pen & Paper to make quick notes.
->Make you mind for observing interesting important items which you see and related to goal
->Try not to interrupt test execution, better to let it finish
->Try long term test execution , when you need to test capacity of your application . I prefer 12/24 hour run over weekends.
->Try mixing up load scenario for different execution, More user, less time, more user long time, constant user increment, user high-low combination(like spike) , combining high load and security scanning etc
This is fully depend on application that you are working with. It may not be based on my experiences. Try to get more load scenarios involving with your application architecture & performance goals.
How to write a performance bug report? :
This is interesting, usually bugs are very specific to UI or user interactions. But, as performance testing is more related to application architecture testing. So, performance bugs should also have reference with application execution as well as how it happened from architectural prospective. That means, this standard procedure of bug reporting and + Something. And This "something" part should be developer friendly as well as full detail with references(as developers will read and get find what you are trying to explain). Let's see more detail on the "Something" with example.
Example , for a web application with http protocol, when you click a button, a function bug will refer to only the use case, but a performance bug should also refer to what request we send to server & received from server in details.
You should add reference on the requested URL, header details, body details. It should have detail explanation on those areas.
For this example, let say a http request Body has 25 parameters & 21 of them with values. So,
- What are those parameters,
- What are the reference of the parameters,
- Why those values are there.
Same justification for request header also. Up to this, it is only description.
So, what will be inside evidence? It should reflect your requirements. Let's say you need to know size, time and capacity for this request and you get the bug. So, in bug report you should provide
a. The total size of the request send to server
b. Time taken for server processing(http 200 response) for this requests
c. How the time varies in case of different load scenario
d. How many responses are received for this request.
e. Individual and total size of all of those responses.
f. Individual and total time of all of those responses for different load scenarios
g.What is the rendering time for those request(Time from individual single user perform the same task in browser – server response time calculated from server)
h. For Rendering, which response takes how much time in browser(from browser tool, or fiddler or proxy)
And, From individual request, you need to break down each parameter detail and possible cause for taking the amount of time that the request is taking.
For advance QAs : (Root Cause Analysis & Suggestions) :
You can go more further, for that you might need to have access of the code. You have to include profiler and should see what cause taking more time for that single request processing.
For web application, use profiler in IIS as well as in code base to pinpoint each request impact. Like, we are doing in our project , tracing from IIS, prove with profiler in IIS, tracing respective event inside code with code analysis tools and provide evidence on what cause the extra time.
Some, senior QAs can go more further. They can provide possible solution to overcome situation.
Example : In our project, we have found , we are not using viewstate(asp.net) properly that cases a huge time. So, we are in need for optimizing view state. But, there are several ways to do. So, we are suggesting different ways as well as impact of the ways for solving. This involve retesting ,that verifies the bug . This part should consider business requirements as well as security requirements.
So , summary, a bug should contains steps to reproduce and standard items mandatory for a function bug , as well as Screenshots, Logs, Charts, test results, Analysis reports, Code snipped, Demo projects, tool references etc.
Thanks...:)