What are the Reports/Listeners in JMeter? Why we use them?
In this article we are going to see what are the reports (Listeners) in JMeter and why we use that. Please see my previous posts for primary idea about jmeter.
Jmeter Listener used to listen and process for suitable views/reports. That needed more memory and processor. Different listener of jmeter performs different purposes.
-Listeners are used to view/save/read from running/saved test results.
-Listeners are processed at the end of the scope in which they are found.
-The results are stored as XML files(".jtl" ext). But storing as CSV is the most efficient option[it is less detailed than XML]
-Listeners do not process sample data in non-GUI mode, but the raw result data will be saved if an output file(configured). We can load that data with any suitable Listener.(From File menu)
Using Tips :
-Clear (Ctrl+Shift+E)/Clear All (Ctrl+E) before loading any result file.
-Configure CSV format before string results from jmeter.properties.
-Use function name inside CSV not variable name for remote running(in client server more, variable does not saved) .
-Use Simple Data Writer, and use the CSV format.
[Note :
Most of the listeners currently keep a copy of every sample in their scope, so they are resource hungry.. These don't need to current data for processing separately, so they don't copy.
-Simple Data Writer
-BeanShell/BSF Listener
-Mailer Visualizer
-Monitor Results
-Summary Report
And these Listeners don't keep copies of every single sample, so less memory needed.
You can see my configuring jmeter posts for having more idea on how to configure jmeter, listeners and all reports.
In the following section, we will discuses more about why we use the particular listener, not about what is that because Jmeter doc provides vast information about that.
Aggregate Graph:
-The aggregate graph is similar to the aggregate report except showing results by bar graphs and save the graph as a PNG file.
-It is better to define the graph settings(spatially following client requirements)
-No settings are saved with jmeter project file.
-Very Useful to see results after a distributed test execution(aggregated results).
-Never Use this while running test as it will cause memory overflow
Aggregate Report:
-Same facility as Simple Data Writer.
-Results are shown in tabular format(request count/min/max/average/error rate/approximate throughput (request/second) and KB/s throughput)
- When "Include group name in label?" is selected, the name of the thread group is added as a prefix to results.
-Very Useful to see results after a distributed test execution(aggregated results).
- Never Use this while running test as it will cause memory overflow
Assertion Results:
- Same facility as Simple Data Writer
- Reports any kind of Assertions fail results with failed sample label.
-Very useful for viewing saved results for validation.
-Never Use this while running test as it will cause memory overflow.
BeanShell Listener:
-BeanShell Listener allows the use of BeanShell for processing samples for saving
-When we need to apply condition(modification) in the execution time, this can be very useful.
Note: I have not used them yet. To know more about BeanShell, see this.
BSF Listener:
-It runs BSF script code to be applied to sample results.
-It is used for processing results in a suitable way(following client requirements)
Note: I have not used this yet.
Comparison Assertion Visualizer:
-It shows the results of any Comparison Assertion element
-Same facility as Simple Data Writer
-It is a little bit resource hungry for the Comparison Assertion element, so try to avoid it
-It is very nice when we need to compare & debug requests under assertion.
Distribution Graph (alpha):
-It will show a bar for every unique response time
-The graph will draw two threshold lines: 50% and 90%.This means, 50% of the response time finished between 0 and the line. Same is true of 90% line.
-Normally, A good application's lines are near in hight. (wild fluctuations occurs for bad application).
-Graphs with no apparent clumping or pattern, may indicate a memory leak, therefore we have to use a profiling tool.
-Same facility as Simple Data Writer
-It is good to view results after test run.
-Never Use this while running test as it will cause memory overflow.
Generate Summary Results:
-It can be placed anywhere
-Main purpose is , Logging or standard output
-Output can be generated after a period of time
-Best uses of this is in multiple test case synchronize running.
-It is mainly intended for batch (non-GUI) runs
Graph Results :
-It shows actual number of requests/minute that the server handle.
-We save result as a file (like simple data writer)
-It also used to view saved data/results.
-We can apply conditions(error/success).
-We can configure the Items to be saved(Press configuration)
-Very Useful to see results after run.
-Never Use this while running test as it will cause memory overflow.
JSR223 Listener:
-It allows JSR223 script code to be applied to sample results.
-It is used for processing results in a suitable way(following client requirements)
Note: I have not used this yet.
Mailer Visualizer:
-It is used for sending mail notification for failed/passed conditions.
-Same facility as Simple Data Writer.
-All settings are mail related settings.
-It is a very good in case of scheduled test plan execution.
-Project stake holders can be notified for test status on error/passing conditions.
-It is best practices to use this
Monitor Results:
-Basically it is a Visualizer for displaying server status
-The "Health" tab shows the status of one or more servers
-The "Performance" shows the performance for one server for the last 1000 samples
-It requires Tomcat 5 or above
-Same facility as Simple Data Writer.
-It takes a lot of resources, so, when ever we need that, we have to specify the interval for monitor samples.(using constant timer).
-In my opinion, It is better to run the test separately(usually occasionally, when we are in a need to update server for better application support) for monitor results as it will monitor the servers rather than application's performance .
Response Time Graph:
-it is a line chart showing the evolution of response time during the test, for each labeled request(every request)
-Same facility as Simple Data Writer.
-It is better to define the graph settings(spatially following client requirements)
-No settings are saved with jmeter project file.
-If necessary , then use this while execution test. This also takes a lot of memory. Better to use this after execution.
Save Responses to a file:
- It can be placed anywhere in the test plan for saving file.
-In sample in its scope, it creates a file of the response Data
- Same facility as Simple Data Writer.
-It is vastly used when we create functional test cases.
-It is very handy when the response data is too large(for currently displaying in tree). It just save them in file.
-It saves file in document type.
-The current sample is saved first, followed by any sub-samples (child samples).
-If a variable name is provided,the names of the files are saved in the order that the sub-samples appear
Simple Data Writer :
-When we save results in our data as CSV/JTI file.
-We can apply conditions(error/success).
-We can configure the Items to be saved(Press configuration)
-It is best practice to use this for storing results which can be viewed after running tests.
Spline Visualizer:
- All options are same as Same as Graph Results but shows a view of all sample divided in 10 segments(10% each)
-It uses spline logic to show a single continuous line.
- It is useful for at a glance graph visibility over graph results.
-Like as Graph Results, Never Use this while running test as it will cause memory overflow.
Summary Report:
-It creates a table row for each differently named request in test.
-It is a summary of view results in table
-It shows average results for separate sampler(EX- http request )
-It is a kind of "Aggregate Report"
-It is good to view results of non distributed test.
- Same facility as Simple Data Writer.
View Results in Table:
-Same facility as View Result Tree but all are shows in tabular format.
-By default, it shows the main (parent) samples, on user checking, it displays the sub-samples (child samples)
-Never Use this while running test as it will cause memory overflow.
View Results Tree:
-View Results Tree shows a tree of all sample & responses in details(HTML/Document/Text/JSON/Regexp Tester/XML format)
- Same facility as Simple Data Writer.
-If there is no content-type provided, then the content will not be displayed in the any of the Response Data panels
-By Default, the response data is larger than 200K, then it won't be displayed.Change the limit from JMeter property view.results.tree.max_size. See this post for how to edit.
-It is the most useful listener for debugging test case. There are various way to use this for optimizing the test plan, view requests, view responses.
-Never Use this while running test as it will cause memory overflow.
-Listeners are used to view/save/read from running/saved test results.
-Listeners are processed at the end of the scope in which they are found.
-The results are stored as XML files(".jtl" ext). But storing as CSV is the most efficient option[it is less detailed than XML]
-Listeners do not process sample data in non-GUI mode, but the raw result data will be saved if an output file(configured). We can load that data with any suitable Listener.(From File menu)
Using Tips :
-Clear (Ctrl+Shift+E)/Clear All (Ctrl+E) before loading any result file.
-Configure CSV format before string results from jmeter.properties.
-Use function name inside CSV not variable name for remote running(in client server more, variable does not saved) .
-Use Simple Data Writer, and use the CSV format.
[Note :
Most of the listeners currently keep a copy of every sample in their scope, so they are resource hungry.. These don't need to current data for processing separately, so they don't copy.
-Simple Data Writer
-BeanShell/BSF Listener
-Mailer Visualizer
-Monitor Results
-Summary Report
And these Listeners don't keep copies of every single sample, so less memory needed.
You can see my configuring jmeter posts for having more idea on how to configure jmeter, listeners and all reports.
In the following section, we will discuses more about why we use the particular listener, not about what is that because Jmeter doc provides vast information about that.
Aggregate Graph:
-The aggregate graph is similar to the aggregate report except showing results by bar graphs and save the graph as a PNG file.
-It is better to define the graph settings(spatially following client requirements)
-No settings are saved with jmeter project file.
-Very Useful to see results after a distributed test execution(aggregated results).
-Never Use this while running test as it will cause memory overflow
Aggregate Report:
-Same facility as Simple Data Writer.
-Results are shown in tabular format(request count/min/max/average/error rate/approximate throughput (request/second) and KB/s throughput)
- When "Include group name in label?" is selected, the name of the thread group is added as a prefix to results.
-Very Useful to see results after a distributed test execution(aggregated results).
- Never Use this while running test as it will cause memory overflow
Assertion Results:
- Same facility as Simple Data Writer
- Reports any kind of Assertions fail results with failed sample label.
-Very useful for viewing saved results for validation.
-Never Use this while running test as it will cause memory overflow.
BeanShell Listener:
-BeanShell Listener allows the use of BeanShell for processing samples for saving
-When we need to apply condition(modification) in the execution time, this can be very useful.
Note: I have not used them yet. To know more about BeanShell, see this.
BSF Listener:
-It runs BSF script code to be applied to sample results.
-It is used for processing results in a suitable way(following client requirements)
Note: I have not used this yet.
Comparison Assertion Visualizer:
-It shows the results of any Comparison Assertion element
-Same facility as Simple Data Writer
-It is a little bit resource hungry for the Comparison Assertion element, so try to avoid it
-It is very nice when we need to compare & debug requests under assertion.
Distribution Graph (alpha):
-It will show a bar for every unique response time
-The graph will draw two threshold lines: 50% and 90%.This means, 50% of the response time finished between 0 and the line. Same is true of 90% line.
-Normally, A good application's lines are near in hight. (wild fluctuations occurs for bad application).
-Graphs with no apparent clumping or pattern, may indicate a memory leak, therefore we have to use a profiling tool.
-Same facility as Simple Data Writer
-It is good to view results after test run.
-Never Use this while running test as it will cause memory overflow.
Generate Summary Results:
-It can be placed anywhere
-Main purpose is , Logging or standard output
-Output can be generated after a period of time
-Best uses of this is in multiple test case synchronize running.
-It is mainly intended for batch (non-GUI) runs
Graph Results :
-It shows actual number of requests/minute that the server handle.
-We save result as a file (like simple data writer)
-It also used to view saved data/results.
-We can apply conditions(error/success).
-We can configure the Items to be saved(Press configuration)
-Very Useful to see results after run.
-Never Use this while running test as it will cause memory overflow.
JSR223 Listener:
-It allows JSR223 script code to be applied to sample results.
-It is used for processing results in a suitable way(following client requirements)
Note: I have not used this yet.
Mailer Visualizer:
-It is used for sending mail notification for failed/passed conditions.
-Same facility as Simple Data Writer.
-All settings are mail related settings.
-It is a very good in case of scheduled test plan execution.
-Project stake holders can be notified for test status on error/passing conditions.
-It is best practices to use this
Monitor Results:
-Basically it is a Visualizer for displaying server status
-The "Health" tab shows the status of one or more servers
-The "Performance" shows the performance for one server for the last 1000 samples
-It requires Tomcat 5 or above
-Same facility as Simple Data Writer.
-It takes a lot of resources, so, when ever we need that, we have to specify the interval for monitor samples.(using constant timer).
-In my opinion, It is better to run the test separately(usually occasionally, when we are in a need to update server for better application support) for monitor results as it will monitor the servers rather than application's performance .
Response Time Graph:
-it is a line chart showing the evolution of response time during the test, for each labeled request(every request)
-Same facility as Simple Data Writer.
-It is better to define the graph settings(spatially following client requirements)
-No settings are saved with jmeter project file.
-If necessary , then use this while execution test. This also takes a lot of memory. Better to use this after execution.
Save Responses to a file:
- It can be placed anywhere in the test plan for saving file.
-In sample in its scope, it creates a file of the response Data
- Same facility as Simple Data Writer.
-It is vastly used when we create functional test cases.
-It is very handy when the response data is too large(for currently displaying in tree). It just save them in file.
-It saves file in document type.
-The current sample is saved first, followed by any sub-samples (child samples).
-If a variable name is provided,the names of the files are saved in the order that the sub-samples appear
Simple Data Writer :
-When we save results in our data as CSV/JTI file.
-We can apply conditions(error/success).
-We can configure the Items to be saved(Press configuration)
-It is best practice to use this for storing results which can be viewed after running tests.
Spline Visualizer:
- All options are same as Same as Graph Results but shows a view of all sample divided in 10 segments(10% each)
-It uses spline logic to show a single continuous line.
- It is useful for at a glance graph visibility over graph results.
-Like as Graph Results, Never Use this while running test as it will cause memory overflow.
Summary Report:
-It creates a table row for each differently named request in test.
-It is a summary of view results in table
-It shows average results for separate sampler(EX- http request )
-It is a kind of "Aggregate Report"
-It is good to view results of non distributed test.
- Same facility as Simple Data Writer.
View Results in Table:
-Same facility as View Result Tree but all are shows in tabular format.
-By default, it shows the main (parent) samples, on user checking, it displays the sub-samples (child samples)
-Never Use this while running test as it will cause memory overflow.
View Results Tree:
-View Results Tree shows a tree of all sample & responses in details(HTML/Document/Text/JSON/Regexp Tester/XML format)
- Same facility as Simple Data Writer.
-If there is no content-type provided, then the content will not be displayed in the any of the Response Data panels
-By Default, the response data is larger than 200K, then it won't be displayed.Change the limit from JMeter property view.results.tree.max_size. See this post for how to edit.
-It is the most useful listener for debugging test case. There are various way to use this for optimizing the test plan, view requests, view responses.
-Never Use this while running test as it will cause memory overflow.
So, best practice, Save the results and see after test execution.
Thanks...:)... If you have more of their uses, please have comments.
Thanks...:)... If you have more of their uses, please have comments.