This is our environment
1.Oracle 8.1.7.4.1--Dont ask me why it was never migrated :)
2.Toplink
3.Tomcat 5.5(Uses JDK 1.5)
4.JDK1.5(Appserver builtin JDK1.5)
5.IIS 6.
We have a HRIS system(OLTP ) providing Employee self service for clients.
OUr IIS and APpserver sits on a same Physical server and DB in a diferent box. To comply with auditing standards we are asked to move the IIS to a DMZ(Which is quite normal but we havent so far) separate the APPServer from IIS.
New set up should be similiar to this
IIS-->AJP(1.3)--Tomcat--Appserver--JDBC--DB
IIS will be redirecting to the TOmcat container and then from Tomcat to APPserver. TOmcat and APpserver sits on a Same Server. Different colors reprsents different boxes.
We purchased Brand new servers to replace all the existing servers except the DB server but the IPs/Names were retained to minimize the complexity during migration.All the applications were migrated without any issue and everything looked fine for 2 weeks but all that was waiting so long to explode finally exploded.
We the concurrent access increases we were simply not able to go beyond the login page(Which is the first page). People who have logged in dont have any problems but the problem lied in login page .
Our appserver makes a set of connection when we startup and then makes new set of connections based on the usage
We tried increasing at the Tomcat MAx number of threads, Timeout,Enable lookss=False. After a week long we couldnt arrive at a conclusion. Appserver CPU and DB server CPU utlization hardly went beyond 10% during the slowness so we came to a conslusion that there is no resource shortage.
When contacted the application team they were simply saying "its all because of migration that you guys did, it was working before why now, so please look at the application setup"
Since the iSsue became very critical Development team as requested to look into the issues,We got the thread dump during the problem which stated the message below
at java.net.Inet4AddressImpl.getHostByAddr(Native Method)
at java.net.InetAddress$1.getHostByAddr(InetAddress.java:842)
at java.net.InetAddress.getHostFromNameService(InetAddress.java:532)
at java.net.InetAddress.getHostName(InetAddress.java:475)
at java.net.InetAddress.getHostName(InetAddress.java:447)
at java.net.InetSocketAddress.getHostName(InetSocketAddress.java:210)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:341)
at java.net.Socket.connect(Socket.java:507)
at java.net.Socket.connect(Socket.java:457)
at java.net.Socket.
at java.net.Socket.
at oracle.net.nt.TcpNTAdapter.connect(Unknown Source)
at oracle.net.nt.ConnOption.connect(Unknown Source)
at oracle.net.nt.ConnStrategy.execute(Unknown Source)
at oracle.net.resolver.AddrResolution.resolveAndExecute(Unknown Source)
at oracle.net.ns.NSProtocol.establishConnection(Unknown Source)
at oracle.net.ns.NSProtocol.connect(Unknown Source)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:706)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.
The first few lines gave some hint the we are having some problems in the DNS server. When we tried NSLOOKUP IP it didnt work but NSLOOKUP with HOSTNAME works so we cam e to a conclusion that reverse dns lookpup fails which matched our error message along with a Metalink doc.
As a fix we added the Other server names in etc/host file of appserver and DB server.
Oh no...There was problem after that and then we came to know that JDBC uses Reverse DNSlookup to make connections.
Happy reading
