Subject: PROG pipe failed: Illegal seek

From: [email-masked]
Date: Tue, 6 Mar 2007 13:53:48 -0800

Our environment is beginning to have a somewhat large Argus config, and
experiencing some stability problems.  Here is our current output from
"argusctl status."

ARGUS/2.0 200 OK
status:    running
version:   3.4_RC-20050419
perlver:   5.8.6
osinfo:    SunOS 5.9 sun4u
objects:   4110
services:  2968
notifies:  1454
uptime:    1d 3:55:26
idle:      61.84% 62.59% 64.15%
monrate:   8.96 8.95 8.89 per second

Lately I am finding that I have a hard time with reloads of the config, or
with starting and stopping Argus.  The first symptom I see is one of the
following two cases in the log file.

[2007/3/3 18:20:03] [18250] successful restart - Argus running
[2007/3/3 18:20:24] [18250] GRAPH pipe failed: Illegal seek
[2007/3/3 18:20:24] [18250] GRAPH pipe failed: Illegal seek

[2007/3/5 9:45:50] [28621] successful restart - Argus running
[2007/3/5 9:45:58] [28621] PROG pipe failed: Illegal seek

In the first case Argus appears to continue running normally after the
error and I don't perceive any ongoing problems.  But in the second case,
Argus either hangs completely, or spawns a new child process, in which case
I end up with 3 (or more) argusd processes running.  In a hup scenario,
this behavior is almost guaranteed, so I have started using stop/start
instead of hup.  It happens almost exclusively at startup and if it
survives the first 30 seconds after startup, it will continue to be fine
until the next restart, though I have had 1 or 2 similar failures during
normal operation.

Does anyone have any clues about where to start looking for the source or
remediation of this problem?

I've looked at the code, and only find 3 instances of this message:
lib/DataBase.pm:        my $m = "pipe failed: $!";
lib/Graph.pm:   my $m = "pipe failed: $!";
lib/Prog.pm:    my $m = "pipe failed: $!";

Prog.pm seems to be the culprit, but I don't understand what would cause a
pipe to fail regularly like this.  We do use Prog a lot for customized
tests.

...Matt Cleveland
...Web Architect
...Health Net Inc
...916.935.1248
[email-masked]

* There's no place like 127.0.0.1

---------------------------------------------------------------------
This message, together with any attachments, is
intended only for the use of the individual or entity
to which it is addressed. It may contain information
that is confidential and prohibited from disclosure.
If you are not the intended recipient, you are hereby
notified that any dissemination or copying of this
message or any attachment is strictly prohibited. If
you have received this message in error, please notify
the original sender immediately by telephone or by
return e-mail and delete this message, along with any
attachments, from your computer. Thank you.

---------------------------------------------------------------------

---

From: Jeff Weisberg <[email-masked]>
Date: Tue, 6 Mar 2007 18:05:53 -0500 (EST)

| [2007/3/5 9:45:58] [28621] PROG pipe failed: Illegal seek
| 
| Does anyone have any clues about where to start looking for the source or
| remediation of this problem?

the error message means:
	you are running solaris
	perl is compiled as a 32 bit app
	you are out of usable file descriptors

32 bit apps under solaris are limited to only 255
file descriptors.

see:
	http://search.cpan.org/~nwclark/perl-5.8.8/README.solaris#Limits_on_Numbers_of_Open_Files_on_Solaris.

the workarounds:

	newer argus code will be faster and has better startup
	behaviour, which may reduce the number of descriptors used.

	run your tests less often, so fewer are running at
	any given time.

	recompile perl as a 64bit app

	install {Net,Free,Open}BSD or Linux

---

From: Jeff Weisberg <[email-masked]>
Date: Wed, 7 Mar 2007 12:24:57 -0500 (EST)

| 32 bit apps under solaris are limited to only 255
| file descriptors.

there appears to be a fix in opensolaris and solaris 11

    http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=1085341
    http://www.opensolaris.org/os/community/on/flag-days/pages/2006042001/

back ports to solaris 9+10 appear to be "coming soon"