Monday, 20 October 2014

"What is the meaning of that error “java.net.SocketTimeoutException: Read timed out" in Neoload

When NeoLoad sends a request and acknowledged by the server, it will wait for the response.  However, in order not to wait an infinite amount of  time, there's a timeout so NeoLoad can give up and raise an error. That's why you got this “Read timed out” error.
By default, during the run time, this timeout is set to 300s (5 minutes). this period of time is usually sufficient for most situations.

Nevertheless, in specific cases, you might not get any data from the server for more than 300s. In that case; here is the procedure to increase the run time timeout in NeoLoad:

  1. Stop NeoLoad
  2. Edit <install_dir>\conf\controller.properties
  3. Go to the [Runtime] section and locate that key: runtime.so.timeout=300000    Change the value to 600000 to set a 600 seconds timeout (it's in milliseconds)
  4. Save
  5. Start NeoLoad

Now, NeoLoad will only alert you the timeout error if it did not receive any data after 600 seconds

Wednesday, 23 July 2014

Top 10 SQL Server Counters for Monitoring SQL Server Performance

1. SQLServer: Buffer Manager: Buffer cache hit ratio

The buffer cache hit ratio counter represents how often SQL Server is able to find data pages in its buffer cache when a query needs a data page. The higher this number the better, because it means SQL Server was able to get data for queries out of memory instead of reading from disk. You want this number to be as close to 100 as possible. Having this counter at 100 means that 100% of the time SQL Server has found the needed data pages in memory. A low buffer cache hit ratio could indicate a memory problem.

2. SQLServer: Buffer Manager: Page life expectancy

The page life expectancy counter measures how long pages stay in the buffer cache in seconds. The longer a page stays in memory, the more likely SQL Server will not need to read from disk to resolve a query. You should watch this counter over time to determine a baseline for what is normal in your database environment. Some say anything below 300 (or 5 minutes) means you might need additional memory.

3. SQLServer: SQL Statistics: Batch Requests/Sec

Batch Requests/Sec measures the number of batches SQL Server is receiving per second. This counter is a good indicator of how much activity is being processed by your SQL Server box. The higher the number, the more queries are being executed on your box. Like many counters, there is no single number that can be used universally to indicate your machine is too busy. Today’s machines are getting more and more powerful all the time and therefore can process more batch requests per second. You should review this counter over time to determine a baseline number for your environment.

4. SQLServer: SQL Statistics: SQL Compilations/Sec

The SQL Compilations/Sec measure the number of times SQL Server compiles an execution plan per second. Compiling an execution plan is a resource-intensive operation. Compilations/Sec should be compared with the number of Batch Requests/Sec to get an indication of whether or not complications might be hurting your performance. To do that, divide the number of batch requests by the number of compiles per second to give you a ratio of the number of batches executed per compile. Ideally you want to have one compile per every 10 batch requests.

5. SQLServer: SQL Statistics: SQL Re-Compilations/Sec

When the execution plan is invalidated due to some significant event, SQL Server will re-compile it. The Re-compilations/Sec counter measures the number of time a re-compile event was triggered per second. Re-compiles, like compiles, are expensive operations so you want to minimize the number of re-compiles. Ideally you want to keep this counter less than 10% of the number of Compilations/Sec.

6. SQLServer: General Statistics: User Connections

The user connections counter identifies the number of different users that are connected to SQL Server at the time the sample was taken. You need to watch this counter over time to understand your baseline user connection numbers. Once you have some idea of your high and low water marks during normal usage of your system, you can then look for times when this counter exceeds the high and low marks. If the value of this counter goes down and the load on the system is the same, then you might have a bottleneck that is not allowing your server to handle the normal load. Keep in mind though that this counter value might go down just because less people are using your SQL Server instance.

7. SQLServer: Locks: Lock Waits / Sec: _Total

In order for SQL Server to manage concurrent users on the system, SQL Server needs to lock resources from time to time. The lock waits per second counter tracks the number of times per second that SQL Server is not able to retain a lock right away for a resource. Ideally you don't want any request to wait for a lock. Therefore you want to keep this counter at zero, or close to zero at all times.

8. SQLServer: Access Methods: Page Splits / Sec

This counter measures the number of times SQL Server had to split a page when updating or inserting data per second. Page splits are expensive, and cause your table to perform more poorly due to fragmentation. Therefore, the fewer page splits you have the better your system will perform. Ideally this counter should be less than 20% of the batch requests per second.

9. SQLServer: General Statistic: Processes Block

The processes blocked counter identifies the number of blocked processes. When one process is blocking another process, the blocked process cannot move forward with its execution plan until the resource that is causing it to wait is freed up. Ideally you don't want to see any blocked processes. When processes are being blocked you should investigate.

10. SQLServer: Buffer Manager: Checkpoint Pages / Sec

The checkpoint pages per second counter measures the number of pages written to disk by a checkpoint operation. You should watch this counter over time to establish a baseline for your systems. Once a baseline value has been established you can watch this value to see if it is climbing. If this counter is climbing, it might mean you are running into memory pressures that are causing dirty pages to be flushed to disk more frequently than normal.

Monday, 23 June 2014

Understanding Context Switches

Processes contain "threads" that are doing the work; they are scheduled and run on the system CPU (at one CPU not on all available CPU's). A process can have multiple threads but only one thread can run on a CPU at a time. The amount of time a thread runs is called "quatum" and when the time is over the system "switches" to the next thread in line (This is the normal case for a switch) – a "context switch" happened.

If the performance counter shows high context switches, it means that threads have less time to do their work and the system performance might go down. At that time the Citrix Resource Manager or any other monitor will raise an alert to inform the Administrator that something is wrong.

Two other definition of context switches:


Microsoft


"The average rate per second at which the processor switches context among threads. A high rate can indicate that many threads are contending for processor time."

Windows Internals

"When Windows selects a new thread to run, it performs a context switch to it. A context switch is the procedure of saving the volatile state associated with a running thread, loading another thread’s volatile state, and starting the new thread’s execution."


Cause of high Context Switches

Common issue I have encountered is a too small page file or where the page file could dynamically grow (start- and end size not set to the same value). Also an option is the write cache of a (RAID) controller that you might want to change using Microsoft's dskcache utility (or the vendor tool). High activity rates can also result from inefficient hardware or poorly designed applications.

Troubleshooting Context Switches

As always there are different ways to troubleshoot such problem but the main target is to find the process(es) that are generating high context switches. Keep in mind that you might need better hardware.

Now how can I find the amount context switches on the system? The answer is the Microsoft performance counter (perfmon.msc) under system/context switches or thread/context switches.
Looking at the performance monitor for context switches based on threads is hard to figure out what process(es) is causing the high rate.

A better utility is sysinternals process explorer. By default process explorer doesn't show context switches and needs to be set in view | select columns | Process Performance | Activate context switches and context switch delta.
Image  Image
You should see both columns in the main view of process explorer. The context switches row shows the total number of switches since the system boot time. Sort the row and look for a)high values and b) fast growing values both are good indicator for high switch rates of the process(es).
Image
Next check the CSwitch Delta row for a high value, since the value shows the context switches made per process explorer refresh interval (if the "update speed" is set to one second, then you have Context Switches / Sec). Once you have found the process(es) you should find out why the process is generating those context switches.
Image
Values for "bad" Context Switches / sec

The default context switches "red alert" value for Citrix Resource Manager is 14.000 but is for a single CPU. The value is per CPU and if the system has two CPU’s you should change the value to 28.000 or 42.000 for three CPU's and 56.000 for a quad CPU system. Still these values are just some basic suggestions and for a "good" value you have to monitor your system over time.

Thursday, 1 May 2014

What is concurrent users and simultaneous users ?

For example, axis bank application has below transactions

                           1. Login
                    2. Account summary
----------------------------Transfer money Action -------------
                    3. Click Transfer funds
                    4. Select "Beneficiary"
                    5. Enter "Amount"
                    6. Transfer
                    7. Enter "User id" and  Password and click "Confirm"
----------------------------Transfer Money Action end----------
                     8.Logout

So we have 8 transaction for this application and we have one Action (Transfer Money is one action but in that action you can have 5 transaction).

Concurrency : Performing same activity at a time by all users.Mean 100 users are login into the application at a time. But simultaneous users are login into the application one by one. Concurrency has various levels
  • Transaction concurrency 
  • Business Process Concurrency 
  • Application Concurrency 

Transaction concurrency :

We need to test the application with 100 concurrent users, Means "All users(100 users) should perform the same transaction (login or   Account summary .... or  Logout) at a time" this is called transaction level concurrency.

Business Process Concurrency:

We need to test the application with 100 concurrent users for Transfer money Business action, means  "All users(100 users) should perform the same Transfer money action (here may be 30 users are clicking "transfer funds, 20 users are on selecting "Beneficiary", 20 users are in enter "Amount" page,10 users are in "Transfer" page  and 20 users are Enter "User id" and  Password and click "Confirm" ). 

Overall 100 users should be in transfer money action.

Application Concurrency :

All 100 users should perform different transaction or same transaction but at any time 100 user should be accessing the application.

Simultaneous users: All 100 users are login into the application one by one.

Monday, 21 April 2014

Using JConsole to monitor a remote WildFly server

The first thing you need to do is setting up a management user on the remote WildFly server using the add-user.sh/add-user.cmdscript:
What type of user do you wish to add?
a) Management User (mgmt-users.properties)
b) Application User (application-users.properties)
(a): a
Enter the details of the new user to add.
Using realm 'ManagementRealm' as discovered from the existing property files.
Username : administrator
Password requirements are listed below. To modify these restrictions edit the add-
user.properties configuration file.
. . . .
Password :
Re-enter Password :
      
Once created the user, startup the remote WildFly server, specifying a management interface IP or host binding:
standalone.bat -Djboss.bind.address.management=192.168.1.78 
Now, on the local machine, locate the jconsole.sh/jconsole.bat script which is in the JBOSS_HOME/bin folder and execute it. You need to specify the http-remoting.jmx protocol, the hostname and the management port which is 9990. For example in our case you need to specify as Remote Process: service:jmx:http-remoting-jmx://192.168.1.78:9990
Next enter the management Username and Password that you have created. 

jconsole wildfly monitor application server
Click on Connect. We are connected to the remote application server:
jconsole wildlfy application server monitor

JBoss Application Server Tuning Tips

JBoss tuning tip 1: Tune the garbage collector

One strength of the J2SE platform is that it shields the developer from the complexity of memory allocation. However, once garbage collection is the principal bottleneck, it is worth understanding some aspects of this hidden implementation
An object is considered garbage when it can no longer be reached from any pointer in the running program. The most straightforward garbage collection algorithms simply iterate over every reachable object. Any objects left over are then considered garbage. The time this approach takes is proportional to the number of live objects,

The complete address space reserved for object memory can be divided into the young and tenured generations.

The young generation consists of eden and two survivor spaces. Most objects are initially allocated in eden. One survivor space is empty at any time, and serves as the destination of any live objects in eden and the other survivor space during the next copying collection. Objects are copied between survivor spaces in this way until they are old enough to be tenured (copied to the tenured generation).
A third generation closely related to the tenured generation is the permanent generation which holds data needed by the virtual machine to describe objects that do not have an equivalence at the Java language level. For example objects describing classes and methods are stored in the permanent generation

Use the the command line option -verbose:gc causes information about the heap and garbage collection to be printed at each collection. For example, here is output from a large server application:
jboss performance tuning It's demonstrated that an application that spends 10% of its time in garbage collection can lose 75% of its throughput when scaled out to 32 processors
(http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html)

JBoss tuning tip 2: Set -Xms and -Xmx to the same value

By default, the virtual machine grows or shrinks the heap at each collection to try to keep the proportion of free space to live objects at each collection within a specific range.
Setting -Xms and -Xmx to the same value. This increase predictability by removing the most important sizing decision from the virtual machine.

JBoss tuning tip 3: Use server VM

The server JVM is better suited to longer running applications. To enable it simply set the -server option on the command line.

JBoss tuning tip 4:  Turn off distributed gc


The RMI system provides a reference counting distributed garbage collection algorithm. This system works by having the server keep track of which clients have requested access to remote objects running on the server. When a reference is made, the server marks the object as "dirty" and when a client drops the reference, it is marked as being "clean.". However this system is quite expensive and by default runs every minute.

Set it to run every 30 minute at least
-Dsun.rmi.dgc.client.gcInterval=1800000
-Dsun.rmi.dgc.server.gcInterval=1800000

JBoss tuning tip 5:  Turn on parallel gc


If you have multiple proessors you can do your garbage collection with multiple threads. By default the parallel collector runs a collection thread per processor, that is if you have an 8 processor box then you'll garbage collect your data with 8 threads. In order to turn on the parallel collector use the flag -XX:+UseParallelGC. You can also specify how many threads you want to dedicate to garbage collection using the flag -XX:ParallelGCThreads=8.

JBoss tuning tip 6: Don't use Huge heaps, use a cluster


More JVMs/smaller heaps can outperform fewer JVMs/Larger Heaps. So instead of huge heaps, use additional server nodes. Set up a JBoss cluster and balance work between nodes.

JBoss tuning tip 7: Don't choose an heap larger then 70% of your OS memory

Choose a maximum heap size not more then 70% of the memory to avoid excessive page faults and thrashing.

JBoss tuning tip 8:  Tune the Heap ratio


This is one of most important tuning factor: the heap ratio. The heap ratio specifies how the amount of the total heap will be partitioned between the young and the tenured space.  What happens if you have lots of long lived data (cached data, collections ) ? maybe you're in this situation:

jboss performance tuning 
The problem here is that the long lived data overflows the tenured generation. When a collection is needed the tenured generation is basically full of live data. Much of the young generation is also filled with long lived data. The result was that a minor collection could not be done successfully (there wasn't enough room in the tenured generation for the anticipated promotions out of the young generation) so a major collection was done.
The major collection worked fine, but the results again was that the tenured generation was full of long lived data and there was long lived data in the young generation. There was also free space in the young generation for more allocations, but the next collection was again destined to be a major collection.

This will eventually bring your application to crawl !!!!!
By decreasing the space in the young generation and putting that space into the tenured generation (a value of NewRatio larger than the default value was chosen), there was enough room in the tenured generation to hold all the long lived data and also space to support minor collections. This particular application used lots of short lived objects so after the fix mostly minor collections were done.
NewRatio is a flag that specifies the amount of the total heap that will be partitioned into the young generation. It's the tenured-generation-size / young-generation-size. For example, setting -XX:NewRatio=3 means that the ratio between the young and tenured generation is 1:3

If you want a more precise control over the young generation : NewSize is the initial size of the young generation, MaxNewSize will specify the maximum size of the young generation
What is the recommeded heap ratios Set the tenured generation to be approximately two times the size of the young generation. With a 2GB of RAM the recommended sizes are 1200MB for the heap and 400MB for the young generation. This recommendation is only a starting point, you have to tune from there and to do that you have to gather and analyze the garbage collection statistics.
JBoss thread pool is defined into conf/jboss-service.xml
01.<mbean code="org.jboss.util.threadpool.BasicThreadPool"
02.name="jboss.system:service=ThreadPool">
03.<attribute name="Name">JBoss System Threads</attribute>
04.<attribute name="ThreadGroupName">System Threads</attribute>
05.<attribute name="KeepAliveTime">60000</attribute>
06.<attribute name="MaximumPoolSize">10</attribute>
07.<attribute name="MaximumQueueSize">1000</attribute>
08.<attribute name="BlockingMode">run</attribute>
09.</mbean>
For most applications this defaults will just work well, however if you are running an application with issues lots of requests to jboss (such as EJB invocations) then monitor your thread pool. Open the Web Console and look for the MBean jboss.system:service=ThreadPool.

 jboss performance tuning
Start a monitor on the QueueSize parameter. Have you got a QueueSize which reaches MaximumPoolSize ? then probably you need to set a higher MaximumPoolSize pool size attribute
Watchout! Speak at first with your sysadmin and ensure that the CPU capacity support the increase in threads.


Watchout! if your threads make use of JDBC connections you'll probably need to increase also the JDBC connection pool accordingly. Also verify that your HTTP connector is enabled to handle that amount of requests


JBoss tuning tip: Check the 
Embedded web container

JBoss supports connectors for http, https, and ajp. The configuration file is server.xml and it's deployed in the root of JBoss web container (In JBoss 4.2.0 it's:  "JBOSS_HOME\server\default\deploy\jboss-web.deployer")
1.<Connector port="8080" address="${jboss.bind.address}"   
2.maxThreads="250" maxHttpHeaderSize="8192"
3.emptySessionPath="true" protocol="HTTP/1.1"
4.enableLookups="false" redirectPort="8443" acceptCount="100"
5.connectionTimeout="20000" disableUploadTimeout="true" />
The underlying HTTP connector of JBoss needs to be fine tuned for production settings.  The important parameters are:
maxThreads - This indicates the maximum number of threads to be allocated for handling client HTTP requests. This figure corresponds to the concurrent users that are going to access the application. Depending on the machine configuration, there is a physical limit beyond which you will need to do clustering.
acceptCount - This is the number of request threads that are put in request queue when all available threads are used. When this exceeds, client machines get a request timeout response.
compression - If you set this attribute to “force”, the content will be compressed by JBoss and will be send to browser. Browser will extract it and display the page on screen. Enabling compression can substantially reduce bandwidth requirements of your application.

So how do you know if it's necessary to raise your maxThreads number ? again open the web console and look for the MBean jboss.web:name=http-127.0.0.1-8080,type=ThreadPool. The key attribute is currentThreadsBusy. If it's about 70-80% of the the maxThreads you should consider raising the number of maxThreads. 

jboss performance tuning
 Watch out! if you increase the maxThreads count you need to raise your JBoss Thread pool accordingly.

JBoss tuning tip :  Turn off JSP Compilation in production


JBoss application server regularly checks whether a JSP requires compilation to a servlet before executing a JSP. In a production server, JSP files won’t change and hence you can configure the settings for increased performance.
Open the web.xml in deploy/jboss-web.deployer/conf folder. Look for the jsp servlet in the file and modify the following XML fragment as given below:
<init-param>
2.<param-name>development</param-name>
3.<param-value>false</param-value>
4.</init-param>
5.<init-param>
6.<param-name>checkInterval</param-name>
7.<param-value>300</param-value>
8.</init-param>

Tip 9: Use Prepared Statement Cache:

 In JBoss, by default,prepared statements are not cached. To improve performance one can configure a prepared statement cache of an arbitrary size. You can use in your -ds.xml file the <prepared-statement-cache-size> : this is the number of prepared statements per connection to be kept open and reused in subsequent requests.

Tip 10: Remove services you don't need

 JBoss ships with lots of services, however you'll seldom need to use them all. The service is usually deployed as *-service.xml under the deploy directory. Sometimes it's also deployed as .sar/.rar archive.
In order to remove the service, remove the file in the "Server/deploy" column. If needed remove also the relative libs stated under "Server/lib
Servizio
Server/deploy
Server/lib
Mail service
mail-service.xml
mail-plugin.jar, mail.jar,activation.jar
Cache invalidation service
cache-invalidation-service.xml
 
J2EE client deployer service
client-deployer-service.xml
 
Hibernate HAR support
hibernate-deployer-service.xml
jboss-hibernate.jar, hibernate2.jar, cglib-full-2.0.1.jar, odmg-3.0.jar
HSQL DB
hsqldb-ds.xml
hsqldb-plugin.jar, hsqldb.jar
Default JMS Service
jms folder
jbossmq.jar
HTTP Invoker (tunnels RMI through HTTP)
http-invoker.sar
 
XA Datasources
jboss-xa-jdbc.rar
 
JMX Console
jmx-console.war
 
Web Console
management/web-console.war
 
JSR-77
management/console-mgr.sar
 
Monitoring mail alerts
monitoring-service.xml
jboss-monitoring.jar
Schedule Manager
schedule-manager-service.xml
scheduler-plugin.jar, scheduler-plugin-example.jar
Sample Schedule service
scheduler-service.xml
 

If you are removing a core JBoss service like JMS or EAR Deployer then you need to remove it also from the jboss-service.xml :
01.<mbean code="org.jboss.management.j2ee.LocalJBossServerDomain"
02.name="jboss.management.local:j2eeType=J2EEDomain,name=Manager">
03.<attribute name="MainDeployer">jboss.system:service=MainDeployer</attribute>
04.<attribute name="SARDeployer">jboss.system:service=ServiceDeployer</attribute>
05.<attribute name="EARDeployer">jboss.j2ee:service=EARDeployer</attribute>
06.<attribute name="EJBDeployer">jboss.ejb:service=EJBDeployer</attribute>
07.<attribute name="RARDeployer">jboss.jca:service=RARDeployer</attribute>
08.<attribute name="CMDeployer">jboss.jca:service=ConnectionFactoryDeployer</attribute>
09.<attribute name="WARDeployer">jboss.web:service=WebServer</attribute>
10.<attribute name="CARDeployer">jboss.j2ee:service=ClientDeployer</attribute>
11.<attribute name="MailService">jboss:service=Mail</attribute>
12.<attribute name="JMSService">jboss.mq:service=DestinationManager</attribute>
13.<attribute name="JNDIService">jboss:service=Naming</attribute>
14.<attribute name="JTAService">jboss:service=TransactionManager</attribute>
15.<attribute name="UserTransactionService">jboss:service=ClientUserTransaction</attribute>
16.<attribute name="RMI_IIOPService">jboss:service=CorbaORB</attribute>
17.</mbean>
Simply comment the attribute relative to the unwanted service

Tip 11: Tell log4j to shut up !

Wel not really anyway Log4j uses a valuable amount of time/CPU so you had better remove unnecessary logs, for example :
  • Remove logs from Console
Comment the following line in log4j.xml in order to remove logs on the Console:
1.<root>
2.<!-- <appender-ref ref=CONSOLE"></appender-ref>  -->
3.<appender-ref ref="FILE"></appender-ref>
4.</root>

remove also the relative appender:
<appender name="CONSOLE" class="org.apache.log4j.ConsoleAppender">
 ....
</appender>
  • Raise log priority
Consider raising the log level to the highest level possible in production. Here only error logs are written:
1.<root>
2.<priority value="ERROR" ></priority>
3.<!--<appender-ref ref="CONSOLE"></appender-ref> -->
4.<appender-ref ref="FILE"></appender-ref>
5.</root>