weird TCP timeout problem

[Matt.A.Cleveland at healthnet.com]

Hi, I've been using Argus for quite some time and it is a raging success at
our company.  Recently I'm having a problem with TCP/URL tests where I am
getting alarms in the middle of the night that as far as I can tell are not
correct, but I also cannot determine the reason for the alarm.  I cannot
tell if this is an Argus problem or an undiagnosed problem with our system
or network, and I am looking for any insight someone might provide to help
narrow this down.

- The test is a TCP/URL test
- Argus reports the problem as - Service DOWN - TCP timeout: expecting
- But the access log on the server shows that the request was served
successfully as far as the web server is concerned (actually it's a direct
request to WebLogic with no web server in the middle)
- Output from the debug log is pasted below
- the timeout is set to 30
- The other very weird thing is that it only fails on this test and one
other.  I am running several other TCP/URL checks against the same server
and these never fail in this way.  So it is hard for me to blame a
networking issue, because it should be more pervasive.

Are there any other diagnostics I can use in Argus?  From the debug log it
appears that the connection is made, the request is sent and some data is
read, but then it times out before getting the EOF.  Can I turn on some
sort of debugging to see exactly which data is read?  Any other ideas of
what would cause this behavior?


DEBUG -
Top:PRD:www.healthnet.com:Connectivity:Transaction_Repository:app2_Tibco
_Claim_Query - [] Service start
DEBUG -
Top:PRD:www.healthnet.com:Connectivity:Transaction_Repository:app2_Tibco
_Claim_Query - [7] TCP Start: connecting - tcp/7201, web-prdapp2, try 1
DEBUG -
Top:PRD:www.healthnet.com:Connectivity:Transaction_Repository:app2_Tibco
_Claim_Query - [7] TCP - connected
DEBUG -
Top:PRD:www.healthnet.com:Connectivity:Transaction_Repository:app2_Tibco
_Claim_Query - [7] TCP - wrote 460 bytes of 460
DEBUG -
Top:PRD:www.healthnet.com:Connectivity:Transaction_Repository:app2_Tibco
_Claim_Query - [7] TCP - read data 303
DEBUG -
Top:PRD:www.healthnet.com:Connectivity:Transaction_Repository:app2_Tibco
_Claim_Query - [7] Service DOWN - TCP timeout: expecting
DEBUG -
Top:PRD:www.healthnet.com:Connectivity:Transaction_Repository:app2_Tibco
_Claim_Query - [7] Service - retrying
DEBUG -
Top:PRD:www.healthnet.com:Connectivity:Transaction_Repository:app2_Tibco
_Claim_Query - [] Service done


Just for fun, here is the debug output from the same test when it succeeds.

DEBUG -
Top:PRD:www.healthnet.com:Connectivity:Transaction_Repository:app2_Tibco
_Claim_Query - [] Service start
DEBUG -
Top:PRD:www.healthnet.com:Connectivity:Transaction_Repository:app2_Tibco
_Claim_Query - [8] TCP Start: connecting - tcp/7201, web-prdapp2, try 1
DEBUG -
Top:PRD:www.healthnet.com:Connectivity:Transaction_Repository:app2_Tibco
_Claim_Query - [8] TCP - connected
DEBUG -
Top:PRD:www.healthnet.com:Connectivity:Transaction_Repository:app2_Tibco
_Claim_Query - [8] TCP - wrote 460 bytes of 460
DEBUG -
Top:PRD:www.healthnet.com:Connectivity:Transaction_Repository:app2_Tibco
_Claim_Query - [8] TCP - read data 303
DEBUG -
Top:PRD:www.healthnet.com:Connectivity:Transaction_Repository:app2_Tibco
_Claim_Query - [8] TCP - read eof
DEBUG -
Top:PRD:www.healthnet.com:Connectivity:Transaction_Repository:app2_Tibco
_Claim_Query - [8]  TEST value HTTP/1.1 200 OK
Date: Wed, 28 Sep 2005 14:59:33 GMT
Server: WebLogic Server 8.1 SP3 Tue Jun 29 23:11:19 PDT 2004 404973
Content-Length: 16
Content-Type: text/plain
Set-Cookie:
JSESSIONID=D6vV06QKwhzCp2q2xJq1xrtrY1FGTHmlLzBqSv8cwygHRC5W21Cp!1758
733194; path=/
Connection: Close

check succeeded

DEBUG -
Top:PRD:www.healthnet.com:Connectivity:Transaction_Repository:app2_Tibco
_Claim_Query - [8]  TEST - curr HTTP/1.1 200 OK
Date: Wed, 28 Sep 2005 14:59:33 GMT
Server: WebLogic Server 8.1 SP3 Tue Jun 29 23:11:19 PDT 2004 404973
Content-Length: 16
Content-Type: text/plain
Set-Cookie:
JSESSIONID=D6vV06QKwhzCp2q2xJq1xrtrY1FGTHmlLzBqSv8cwygHRC5W21Cp!1758
733194; path=/
Connection: Close

check succeeded

DEBUG -
Top:PRD:www.healthnet.com:Connectivity:Transaction_Repository:app2_Tibco
_Claim_Query - [8] Service - UP
DEBUG -
Top:PRD:www.healthnet.com:Connectivity:Transaction_Repository:app2_Tibco
_Claim_Query - [] Service done

::
:: Matt Cleveland
:: Web Architect
:: Health Net Inc.
:: 916.935.1248
:: matt.cleveland at healthnet.com
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::



This  message,together  with  any  attachments, is
intended only for the use of the individual or entity
to which it is addressed. It may contain information
that is confidential and prohibited from disclosure.
If you are not the intended recipient, you are hereby
notified that  any dissemination  or copying of this
message or any attachment is strictly prohibited. If
you have received this message in error, please notify
the  original  sender immediately by telephone or by
return e-mail and delete this message, along with any
attachments, from your computer.  Thank you.