too many open files

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

too many open files

Ray V-2
I started getting this from time to time.  We're
currently dealing with it by restarting hudson (on
tomcat) and then forcing a new build.

It's probably related to my project's build behaving
badly rather than a bug in hudson.  Just thought I'd
mention it since cruisecontrol and local builds have
never had this problem for us.

FATAL: command execution failed
java.io.IOException: java.io.IOException: Too many
open files
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
        at java.lang.ProcessImpl.start(ProcessImpl.java:65)
        at
java.lang.ProcessBuilder.start(ProcessBuilder.java:451)
        at java.lang.Runtime.exec(Runtime.java:591)
        at hudson.Proc$LocalProc.<init>(Proc.java:79)
        at
hudson.Launcher$LocalLauncher.launch(Launcher.java:167)
        at hudson.Launcher.launch(Launcher.java:82)
        at hudson.Launcher.launch(Launcher.java:70)
        at hudson.tasks.Ant.perform(Ant.java:113)
        at
hudson.model.Build$RunnerImpl.build(Build.java:149)
        at
hudson.model.Build$RunnerImpl.doRun(Build.java:121)
        at
hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:121)
        at hudson.model.Run.run(Run.java:559)
        at hudson.model.Build.run(Build.java:101)
        at hudson.model.Executor.run(Executor.java:59)



       
____________________________________________________________________________________Looking for a deal? Find great prices on flights and hotels with Yahoo! FareChase.
http://farechase.yahoo.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: too many open files

Kohsuke Kawaguchi
Administrator

Actually, your build runs as a separate process, so no matter what it
does it shouldn't cause this problem.

What Unix do you use? I wodner if /proc exposes a list of files that are
open.

Ray V wrote:

> I started getting this from time to time.  We're
> currently dealing with it by restarting hudson (on
> tomcat) and then forcing a new build.
>
> It's probably related to my project's build behaving
> badly rather than a bug in hudson.  Just thought I'd
> mention it since cruisecontrol and local builds have
> never had this problem for us.
>
> FATAL: command execution failed
> java.io.IOException: java.io.IOException: Too many
> open files
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> at
> java.lang.ProcessBuilder.start(ProcessBuilder.java:451)
> at java.lang.Runtime.exec(Runtime.java:591)
> at hudson.Proc$LocalProc.<init>(Proc.java:79)
> at
> hudson.Launcher$LocalLauncher.launch(Launcher.java:167)
> at hudson.Launcher.launch(Launcher.java:82)
> at hudson.Launcher.launch(Launcher.java:70)
> at hudson.tasks.Ant.perform(Ant.java:113)
> at
> hudson.model.Build$RunnerImpl.build(Build.java:149)
> at
> hudson.model.Build$RunnerImpl.doRun(Build.java:121)
> at
> hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:121)
> at hudson.model.Run.run(Run.java:559)
> at hudson.model.Build.run(Build.java:101)
> at hudson.model.Executor.run(Executor.java:59)

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: too many open files

Kohsuke Kawaguchi
Administrator
Kohsuke Kawaguchi wrote:
> Actually, your build runs as a separate process, so no matter what it
> does it shouldn't cause this problem.
>
> What Unix do you use? I wodner if /proc exposes a list of files that are
> open.

Yeah, try "ls -la /proc/<pid>/fd"

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: too many open files

Kohsuke Kawaguchi
Administrator
Kohsuke Kawaguchi wrote:
> Kohsuke Kawaguchi wrote:
>> Actually, your build runs as a separate process, so no matter what it
>> does it shouldn't cause this problem.
>>
>> What Unix do you use? I wodner if /proc exposes a list of files that are
>> open.
>
> Yeah, try "ls -la /proc/<pid>/fd"

Or "lsof"

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: too many open files

Ray V-2
Thanks for the quick replies.  I tried lsof on the
hudson process during a build and it got pretty high.
The max on the box (redhat) is 1024 I think.  We may
bump that up.

But you're saying that the hudson proc will start a
new process for each build right?  That means that my
build isn't causing the problem I guess... could it be
the last step where it copies the artifacts?

I removed the javadoc documentation temporarily from
my config to see if that helps this out.

--- Kohsuke Kawaguchi <[hidden email]>
wrote:

> Kohsuke Kawaguchi wrote:
> > Kohsuke Kawaguchi wrote:
> >> Actually, your build runs as a separate process,
> so no matter what it
> >> does it shouldn't cause this problem.
> >>
> >> What Unix do you use? I wodner if /proc exposes a
> list of files that are
> >> open.
> >
> > Yeah, try "ls -la /proc/<pid>/fd"
>
> Or "lsof"
>
> --
> Kohsuke Kawaguchi
> Sun Microsystems                  
> [hidden email]
>



       
____________________________________________________________________________________Be a better Globetrotter. Get better travel answers from someone who knows. Yahoo! Answers - Check it out.
http://answers.yahoo.com/dir/?link=list&sid=396545469

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: too many open files

Kohsuke Kawaguchi
Administrator
Ray V wrote:
> Thanks for the quick replies.  I tried lsof on the
> hudson process during a build and it got pretty high.
> The max on the box (redhat) is 1024 I think.  We may
> bump that up.

Make sure that you are getting the list from the hudson process that
runs the webapp, not the child processes that are doing the build.

> But you're saying that the hudson proc will start a
> new process for each build right?  That means that my
> build isn't causing the problem I guess... could it be
> the last step where it copies the artifacts?

Somewhere Hudson must be forgetting to close file handles. We just need
to find out where, and for that lsof is helpful.

If you can share with us what kind of files that Hudson is opening, that
should be a good clue for me to tell where the leak might be happening.

If you can't share it with public, please consider sending it to me
privately.

> I removed the javadoc documentation temporarily from
> my config to see if that helps this out.

Thanks.

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: too many open files

Ray V-2
Cool, thanks.  Below is the info I collected from
forcing a build (one that failed with this problem)
after the last time I saw this happen.  Next time I'll
try to get a more detailed log (containing the actual
open files).

Like I said it's been happening unpredictably over
time and I suspect the javadocs might be a cause.

Also we're running junit4 unit tests via ant, which
took some strange hacks to get working at all (had to
set fork="false" for some reason on the junit task).
Considering going back to using junit3.  Anyway,
here's some minimal info:

[root@build ~]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) 1024
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 73728
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Random sampling of open files while this last build
was running:
[root@build ~]# lsof | grep java | wc -l 1710
[root@build ~]# lsof | grep hudson | wc -l
17
[root@build ~]# lsof | grep java | wc -l 1710
[root@build ~]# lsof | grep hudson | wc -l
17
[root@build ~]# lsof | grep hudson | wc -l
151
[root@build ~]# lsof | grep hudson | wc -l
16
[root@build ~]# lsof | grep java | wc -l
1844
[root@build ~]# lsof | grep hudson | wc -l 150
[root@build ~]# lsof | grep java | wc -l
1965
[root@build ~]# lsof | grep hudson | wc -l 150
[root@build ~]# lsof | grep hudson | wc -l 150
[root@build ~]# lsof | grep hudson | wc -l 150
[root@build ~]# lsof | grep hudson | wc -l 10

--- Kohsuke Kawaguchi <[hidden email]>
wrote:

> Ray V wrote:
> > Thanks for the quick replies.  I tried lsof on the
> > hudson process during a build and it got pretty
> high.
> > The max on the box (redhat) is 1024 I think.  We
> may
> > bump that up.
>
> Make sure that you are getting the list from the
> hudson process that
> runs the webapp, not the child processes that are
> doing the build.
>
> > But you're saying that the hudson proc will start
> a
> > new process for each build right?  That means that
> my
> > build isn't causing the problem I guess... could
> it be
> > the last step where it copies the artifacts?
>
> Somewhere Hudson must be forgetting to close file
> handles. We just need
> to find out where, and for that lsof is helpful.
>
> If you can share with us what kind of files that
> Hudson is opening, that
> should be a good clue for me to tell where the leak
> might be happening.
>
> If you can't share it with public, please consider
> sending it to me
> privately.
>
> > I removed the javadoc documentation temporarily
> from
> > my config to see if that helps this out.
>
> Thanks.
>
> --
> Kohsuke Kawaguchi
> Sun Microsystems                  
> [hidden email]
>



       
____________________________________________________________________________________Be a better Globetrotter. Get better travel answers from someone who knows. Yahoo! Answers - Check it out.
http://answers.yahoo.com/dir/?link=list&sid=396545469

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: too many open files

Kohsuke Kawaguchi
Administrator
Ray V wrote:
> Cool, thanks.  Below is the info I collected from
> forcing a build (one that failed with this problem)
> after the last time I saw this happen.  Next time I'll
> try to get a more detailed log (containing the actual
> open files).

Thanks.

> Like I said it's been happening unpredictably over
> time and I suspect the javadocs might be a cause.

The code that copies javadoc is using Ant task implementations of <copy>
and <tar>, and the code looks like it is closing the streams properly
(I'd imagine if there's a file handle leak in Ant, it would have been
noticed a long time ago by Ant users.)


> Also we're running junit4 unit tests via ant, which
> took some strange hacks to get working at all (had to
> set fork="false" for some reason on the junit task).
> Considering going back to using junit3.  Anyway,
> here's some minimal info:
>
> [root@build ~]# ulimit -a
> core file size          (blocks, -c) 0
> data seg size           (kbytes, -d) unlimited
> file size               (blocks, -f) unlimited
> pending signals                 (-i) 1024
> max locked memory       (kbytes, -l) 32
> max memory size         (kbytes, -m) unlimited
> open files                      (-n) 1024
> pipe size            (512 bytes, -p) 8
> POSIX message queues     (bytes, -q) 819200
> stack size              (kbytes, -s) 10240
> cpu time               (seconds, -t) unlimited
> max user processes              (-u) 73728
> virtual memory          (kbytes, -v) unlimited
> file locks                      (-x) unlimited
>
> Random sampling of open files while this last build
> was running:
> [root@build ~]# lsof | grep java | wc -l 1710
> [root@build ~]# lsof | grep hudson | wc -l
> 17
> [root@build ~]# lsof | grep java | wc -l 1710
> [root@build ~]# lsof | grep hudson | wc -l
> 17
> [root@build ~]# lsof | grep hudson | wc -l
> 151
> [root@build ~]# lsof | grep hudson | wc -l
> 16
> [root@build ~]# lsof | grep java | wc -l
> 1844
> [root@build ~]# lsof | grep hudson | wc -l 150
> [root@build ~]# lsof | grep java | wc -l
> 1965
> [root@build ~]# lsof | grep hudson | wc -l 150
> [root@build ~]# lsof | grep hudson | wc -l 150
> [root@build ~]# lsof | grep hudson | wc -l 150
> [root@build ~]# lsof | grep hudson | wc -l 10
150 files seem to be a lot to me. I bet some of them must be leak.

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: too many open files

Ray V-2
In reply to this post by Ray V-2
This issue came up again... here's the console output
followed by a list of open files taken after the build
failed... after waiting a while the open files went
back down again by itself.  not sure what else to try.

started
Updating http://svn:81/svn/LDS/LDS30/trunk
U         build.xml
At revision 540
[trunk] $ ant build
FATAL: command execution failed
java.io.IOException: java.io.IOException: Too many
open files
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
        at java.lang.ProcessImpl.start(ProcessImpl.java:65)
        at
java.lang.ProcessBuilder.start(ProcessBuilder.java:451)
        at java.lang.Runtime.exec(Runtime.java:591)
        at hudson.Proc$LocalProc.<init>(Proc.java:79)
        at
hudson.Launcher$LocalLauncher.launch(Launcher.java:167)
        at hudson.Launcher.launch(Launcher.java:82)
        at hudson.Launcher.launch(Launcher.java:70)
        at hudson.tasks.Ant.perform(Ant.java:113)
        at
hudson.model.Build$RunnerImpl.build(Build.java:149)
        at
hudson.model.Build$RunnerImpl.doRun(Build.java:121)
        at
hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:121)
        at hudson.model.Run.run(Run.java:559)
        at hudson.model.Build.run(Build.java:101)
        at hudson.model.Executor.run(Executor.java:59)
ERROR: Failed to archive artifacts:
trunk/build/dist/war/lds.war,trunk/build/dist/lds.jar,trunk/build/dist/lds-dao.jar
hudson.util.IOException2: Failed to copy
/root/.hudson/jobs/lds-30-trunk/workspace/trunk/build/dist/war/lds.war,trunk/build/dist/lds.jar,trunk/build/dist/lds-dao.jar
to
/root/.hudson/jobs/lds-30-trunk/builds/2007-06-13_15-02-00/archive
        at hudson.FilePath$19.invoke(FilePath.java:680)
        at hudson.FilePath$19.invoke(FilePath.java:647)
        at hudson.FilePath.act(FilePath.java:239)
        at hudson.FilePath.copyRecursiveTo(FilePath.java:647)
        at
hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:65)
        at hudson.model.Build$RunnerImpl.post(Build.java:137)
        at hudson.model.Run.run(Run.java:572)
        at hudson.model.Build.run(Build.java:101)
        at hudson.model.Executor.run(Executor.java:59)
Caused by: Failed to copy
/root/.hudson/jobs/lds-30-trunk/workspace/trunk/build/dist/lds.jar
to
/root/.hudson/jobs/lds-30-trunk/builds/2007-06-13_15-02-00/archive/trunk/build/dist/lds.jar
due to java.io.FileNotFoundException
/root/.hudson/jobs/lds-30-trunk/builds/2007-06-13_15-02-00/archive/trunk/build/dist/lds.jar
(Too many open files)
        at
org.apache.tools.ant.taskdefs.Copy.doFileOperations(Copy.java:807)
        at
hudson.FilePath$19$1CopyImpl.doFileOperations(FilePath.java:661)
        at
org.apache.tools.ant.taskdefs.Copy.execute(Copy.java:534)
        at hudson.FilePath$19.invoke(FilePath.java:677)
        ... 8 more
Caused by: java.io.FileNotFoundException:
/root/.hudson/jobs/lds-30-trunk/builds/2007-06-13_15-02-00/archive/trunk/build/dist/lds.jar
(Too many open files)
        at java.io.FileOutputStream.open(Native Method)
        at
java.io.FileOutputStream.<init>(FileOutputStream.java:179)
        at
java.io.FileOutputStream.<init>(FileOutputStream.java:131)
        at
org.apache.tools.ant.types.resources.FileResource.getOutputStream(FileResource.java:213)
        at
org.apache.tools.ant.util.ResourceUtils.copyResource(ResourceUtils.java:379)
        at
org.apache.tools.ant.util.FileUtils.copyFile(FileUtils.java:508)
        at
org.apache.tools.ant.util.FileUtils.copyFile(FileUtils.java:323)
        at
org.apache.tools.ant.taskdefs.Copy.doFileOperations(Copy.java:795)
        ... 11 more
Recording fingerprints
Recording test results
Test reports were found but none of them are new. Did
tests run?
Recording Emma reports
trunk/build/dist/coverage/coverage.xml
FATAL: failed
java.io.FileNotFoundException:
/root/.hudson/jobs/lds-30-trunk/workspace/trunk/build/dist/coverage/coverage.xml
(Too many open files)
        at java.io.FileInputStream.open(Native Method)
        at
java.io.FileInputStream.<init>(FileInputStream.java:106)
        at hudson.FilePath$18.invoke(FilePath.java:608)
        at hudson.FilePath$18.invoke(FilePath.java:606)
        at hudson.FilePath.act(FilePath.java:239)
        at hudson.FilePath.copyTo(FilePath.java:606)
        at hudson.FilePath.copyTo(FilePath.java:594)
        at
hudson.plugins.emma.EmmaPublisher.perform(EmmaPublisher.java:54)
        at hudson.model.Build$RunnerImpl.post(Build.java:137)
        at hudson.model.Run.run(Run.java:572)
        at hudson.model.Build.run(Build.java:101)
        at hudson.model.Executor.run(Executor.java:59)
finished: FAILURE

[rvanderborght@build ~]$ sudo /usr/sbin/lsof | grep
hudson
java      27023          root  mem       REG      
8,2    261710     389933
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/xstream-1.1.3.jar
java      27023          root  mem       REG      
8,2     93395     389932
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/xpp3-1.1.3.3.jar
java      27023          root  mem       REG      
8,2      9766     389931
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/wagon-ssh-external-1.0-alpha-6.jar
java      27023          root  mem       REG      
8,2     32265     389930
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/wagon-ssh-1.0-alpha-7.jar
java      27023          root  mem       REG      
8,2     42727     389929
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/wagon-provider-api-1.0-alpha-6.jar
java      27023          root  mem       REG      
8,2      8336     389928
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/wagon-http-lightweight-1.0-alpha-6.jar
java      27023          root  mem       REG      
8,2      4724     389927
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/wagon-file-1.0-alpha-7.jar
java      27023          root  mem       REG      
8,2   1240894     389926
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/svnkit-1.1.2-hudson-2.jar
java      27023          root  mem       REG      
8,2    111296     389925
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/stapler-1.30.jar
java      27023          root  mem       REG      
8,2     64347     389924
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/remoting-1.105.jar
java      27023          root  mem       REG      
8,2    168568     389923
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/plexus-utils-1.1.jar
java      27023          root  mem       REG      
8,2     13407     389922
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/plexus-interactivity-api-1.0-alpha-4.jar
java      27023          root  mem       REG      
8,2    195394     389921
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/plexus-container-default-1.0-alpha-9.jar
java      27023          root  mem       REG      
8,2     44144     389920
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-settings-2.0.4.jar
java      27023          root  mem       REG      
8,2     19819     389919
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-repository-metadata-2.0.4.jar
java      27023          root  mem       REG      
8,2      5000     389918
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-reporting-api-2.0.4.jar
java      27023          root  mem       REG      
8,2    109543     389917
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-project-2.0.4.jar
java      27023          root  mem       REG      
8,2     30421     389916
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-profile-2.0.4.jar
java      27023          root  mem       REG      
8,2     24084     389915
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-plugin-registry-2.0.4.jar
java      27023          root  mem       REG      
8,2     16392     389914
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-plugin-parameter-documenter-2.0.4.jar
java      27023          root  mem       REG      
8,2     32258     389913
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-plugin-descriptor-2.0.4.jar
java      27023          root  mem       REG      
8,2      8341     389912
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-plugin-api-2.0.4.jar
java      27023          root  mem       REG      
8,2      5738     389911
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-monitor-2.0.4.jar
java      27023          root  mem       REG      
8,2     81575     389910
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-model-2.0.4.jar
java      27023          root  mem       REG      
8,2     14545     389909
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-interceptor-1.105.jar
java      27023          root  mem       REG      
8,2      8658     389908
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-error-diagnostics-2.0.4.jar
java      27023          root  mem       REG      
8,2     20615     389907
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-embedder-2.0.4.jar
java      27023          root  mem       REG      
8,2    137173     389906
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-core-2.0.4.jar
java      27023          root  mem       REG      
8,2     49162     389905
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-artifact-manager-2.0.4.jar
java      27023          root  mem       REG      
8,2     80522     389904
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-artifact-2.0.4.jar
java      27023          root  mem       REG      
8,2      6020     389903
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-agent-1.105.jar
java      27023          root  mem       REG      
8,2     72150     389901
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/logkit-1.0.1.jar
java      27023          root  mem       REG      
8,2    358085     389900
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/log4j-1.2.12.jar
java      27023          root  mem       REG      
8,2     16923     389899
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/jstl-1.1.0.jar
java      27023          root  mem       REG      
8,2    117174     389898
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/jsch-0.1.24.jar
java      27023          root  mem       REG      
8,2   1139113     389897
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/jfreechart-1.0.3.jar
java      27023          root  mem       REG      
8,2    238637     389895
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/jaxen-1.1-beta-11.jar
java      27023          root  mem       REG      
8,2   1800974     389893
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/groovy-all-1.0-jsr-06.jar
java      27023          root  mem       REG      
8,2     47997     389892
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/graph-layouter-1.0.jar
java      27023          root  mem       REG      
8,2    244831     389891
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/ganymed-ssh2-build210.jar
java      27023          root  mem       REG      
8,2      5936     389890
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/doxia-sink-api-1.0-alpha-7.jar
java      27023          root  mem       REG      
8,2    313898     389889
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/dom4j-1.6.1.jar
java      27023          root  mem       REG      
8,2     52915     389888
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-logging-1.1.jar
java      27023          root  mem       REG      
8,2    132239     389887
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-jexl-1.1-hudson-20061106.jar
java      27023          root  mem       REG      
8,2     37430     389886
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-jelly-tags-xml-1.1.jar
java      27023          root  mem       REG      
8,2     22479     389885
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-jelly-tags-fmt-1.0.jar
java      27023          root  mem       REG      
8,2     28630     389884
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-jelly-tags-define-1.0.1-hudson-20061106.jar
java      27023          root  mem       REG      
8,2     83613     389882
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-io-1.3.1.jar
java      27023          root  mem       REG      
8,2     31825     389881
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-fileupload-1.1.jar
java      27023          root  mem       REG      
8,2    139966     389880
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-digester-1.7.jar
java      27023          root  mem       REG      
8,2    165119     389879
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-collections-2.1.jar
java      27023          root  mem       REG      
8,2    188671     389878
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-beanutils-1.7.0.jar
java      27023          root  mem       REG      
8,2     37854     389877
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/classworlds-1.1.jar
java      27023          root  mem       REG      
8,2     63966     389876
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/avalon-framework-4.1.3.jar
java      27023          root  mem       REG      
8,2    443432     389875
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/antlr-2.7.6.jar
java      27023          root  mem       REG      
8,2     11734     389874
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/ant-launcher-1.7.0.jar
java      27023          root  mem       REG      
8,2     92792     389873
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/ant-junit-1.7.0.jar
java      27023          root  mem       REG      
8,2   1289806     389872
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/ant-1.7.0.jar
java      27023          root  mem       REG      
8,2    388826     389902
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/mail-1.4.jar
java      27023          root  mem       REG      
8,2    308394     389896
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/jcommon-1.0.6.jar
java      27023          root  mem       REG      
8,2    852231     389894
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/hudson-core-1.105.jar
java      27023          root  mem       REG      
8,2    174753     389883
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-jelly-1.1-hudson-20061106.jar
java      27023          root  mem       REG      
8,2     62983     389871
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/activation-1.1.jar
java      27023          root 1474w      REG      
8,9      4305   11109289
/home/root/.hudson/jobs/edu-potrero/builds/2007-06-12_12-02-01/changelog.xml
java      27023          root 1704r      REG      
8,9        38   79904958
/home/root/.hudson/jobs/lds-30-trunk/builds/2007-06-13_15-02-00/revision.txt
java      27023          root 2011r      REG      
8,9        38   79905723
/home/root/.hudson/jobs/lds-30-trunk/builds/2007-06-13_17-01-56/revision.txt
java      27023          root 2017w      REG      
8,9       347   79906188
/home/root/.hudson/jobs/lds-30-trunk/builds/2007-06-13_17-01-56/changelog.xml

--- Ray V <[hidden email]> wrote:

> Cool, thanks.  Below is the info I collected from
> forcing a build (one that failed with this problem)
> after the last time I saw this happen.  Next time
> I'll
> try to get a more detailed log (containing the
> actual
> open files).
>
> Like I said it's been happening unpredictably over
> time and I suspect the javadocs might be a cause.
>
> Also we're running junit4 unit tests via ant, which
> took some strange hacks to get working at all (had
> to
> set fork="false" for some reason on the junit task).
>
> Considering going back to using junit3.  Anyway,
> here's some minimal info:
>
> [root@build ~]# ulimit -a
> core file size          (blocks, -c) 0
> data seg size           (kbytes, -d) unlimited
> file size               (blocks, -f) unlimited
> pending signals                 (-i) 1024
> max locked memory       (kbytes, -l) 32
> max memory size         (kbytes, -m) unlimited
> open files                      (-n) 1024
> pipe size            (512 bytes, -p) 8
> POSIX message queues     (bytes, -q) 819200
> stack size              (kbytes, -s) 10240
> cpu time               (seconds, -t) unlimited
> max user processes              (-u) 73728
> virtual memory          (kbytes, -v) unlimited
> file locks                      (-x) unlimited
>
> Random sampling of open files while this last build
> was running:
> [root@build ~]# lsof | grep java | wc -l 1710
> [root@build ~]# lsof | grep hudson | wc -l
> 17
> [root@build ~]# lsof | grep java | wc -l 1710
> [root@build ~]# lsof | grep hudson | wc -l
> 17
> [root@build ~]# lsof | grep hudson | wc -l
> 151
> [root@build ~]# lsof | grep hudson | wc -l
> 16
> [root@build ~]# lsof | grep java | wc -l
> 1844
> [root@build ~]# lsof | grep hudson | wc -l 150
> [root@build ~]# lsof | grep java | wc -l
> 1965
> [root@build ~]# lsof | grep hudson | wc -l 150
> [root@build ~]# lsof | grep hudson | wc -l 150
> [root@build ~]# lsof | grep hudson | wc -l 150
> [root@build ~]# lsof | grep hudson | wc -l 10
>
> --- Kohsuke Kawaguchi <[hidden email]>
> wrote:
>
> > Ray V wrote:
> > > Thanks for the quick replies.  I tried lsof on
> the
> > > hudson process during a build and it got pretty
> > high.
> > > The max on the box (redhat) is 1024 I think.  We
> > may
> > > bump that up.
> >
> > Make sure that you are getting the list from the
> > hudson process that
> > runs the webapp, not the child processes that are
> > doing the build.
> >
> > > But you're saying that the hudson proc will
> start
> > a
> > > new process for each build right?  That means
> that
> > my
> > > build isn't causing the problem I guess... could
> > it be
> > > the last step where it copies the artifacts?
> >
> > Somewhere Hudson must be forgetting to close file
> > handles. We just need
> > to find out where, and for that lsof is helpful.
> >
> > If you can share with us what kind of files that
> > Hudson is opening, that
> > should be a good clue for me to tell where the
> leak
> > might be happening.
> >
> > If you can't share it with public, please consider
> > sending it to me
> > privately.
> >
> > > I removed the javadoc documentation temporarily
> > from
> > > my config to see if that helps this out.
> >
> > Thanks.
> >
> > --
> > Kohsuke Kawaguchi
> > Sun Microsystems                  
> > [hidden email]
> >
>
>
>
>        
>
____________________________________________________________________________________Be
> a better Globetrotter. Get better travel answers
> from someone who knows. Yahoo! Answers - Check it
> out.
>
http://answers.yahoo.com/dir/?link=list&sid=396545469
>
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> [hidden email]
> For additional commands, e-mail:
> [hidden email]
>
>



 
____________________________________________________________________________________
Finding fabulous fares is fun.  
Let Yahoo! FareChase search your favorite travel sites to find flight and hotel bargains.
http://farechase.yahoo.com/promo-generic-14795097

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: too many open files

Kohsuke Kawaguchi
Administrator
Thanks to your report, I was able to fix one place where I forgot to
close. From the list it appears to me that changelog.xml file handle for
write is also leaking, so I take some defensive measure to prevent that,
too.

But looking at your list, of 67 files that are listed, 63 are jar files
in the lib directory, which are obviously not leaks. So in any case it
seems like your need to increase the file handle limit.


Ray V wrote:

> This issue came up again... here's the console output
> followed by a list of open files taken after the build
> failed... after waiting a while the open files went
> back down again by itself.  not sure what else to try.
>
> started
> Updating http://svn:81/svn/LDS/LDS30/trunk
> U         build.xml
> At revision 540
> [trunk] $ ant build
> FATAL: command execution failed
> java.io.IOException: java.io.IOException: Too many
> open files
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> at
> java.lang.ProcessBuilder.start(ProcessBuilder.java:451)
> at java.lang.Runtime.exec(Runtime.java:591)
> at hudson.Proc$LocalProc.<init>(Proc.java:79)
> at
> hudson.Launcher$LocalLauncher.launch(Launcher.java:167)
> at hudson.Launcher.launch(Launcher.java:82)
> at hudson.Launcher.launch(Launcher.java:70)
> at hudson.tasks.Ant.perform(Ant.java:113)
> at
> hudson.model.Build$RunnerImpl.build(Build.java:149)
> at
> hudson.model.Build$RunnerImpl.doRun(Build.java:121)
> at
> hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:121)
> at hudson.model.Run.run(Run.java:559)
> at hudson.model.Build.run(Build.java:101)
> at hudson.model.Executor.run(Executor.java:59)
> ERROR: Failed to archive artifacts:
> trunk/build/dist/war/lds.war,trunk/build/dist/lds.jar,trunk/build/dist/lds-dao.jar
> hudson.util.IOException2: Failed to copy
> /root/.hudson/jobs/lds-30-trunk/workspace/trunk/build/dist/war/lds.war,trunk/build/dist/lds.jar,trunk/build/dist/lds-dao.jar
> to
> /root/.hudson/jobs/lds-30-trunk/builds/2007-06-13_15-02-00/archive
> at hudson.FilePath$19.invoke(FilePath.java:680)
> at hudson.FilePath$19.invoke(FilePath.java:647)
> at hudson.FilePath.act(FilePath.java:239)
> at hudson.FilePath.copyRecursiveTo(FilePath.java:647)
> at
> hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:65)
> at hudson.model.Build$RunnerImpl.post(Build.java:137)
> at hudson.model.Run.run(Run.java:572)
> at hudson.model.Build.run(Build.java:101)
> at hudson.model.Executor.run(Executor.java:59)
> Caused by: Failed to copy
> /root/.hudson/jobs/lds-30-trunk/workspace/trunk/build/dist/lds.jar
> to
> /root/.hudson/jobs/lds-30-trunk/builds/2007-06-13_15-02-00/archive/trunk/build/dist/lds.jar
> due to java.io.FileNotFoundException
> /root/.hudson/jobs/lds-30-trunk/builds/2007-06-13_15-02-00/archive/trunk/build/dist/lds.jar
> (Too many open files)
> at
> org.apache.tools.ant.taskdefs.Copy.doFileOperations(Copy.java:807)
> at
> hudson.FilePath$19$1CopyImpl.doFileOperations(FilePath.java:661)
> at
> org.apache.tools.ant.taskdefs.Copy.execute(Copy.java:534)
> at hudson.FilePath$19.invoke(FilePath.java:677)
> ... 8 more
> Caused by: java.io.FileNotFoundException:
> /root/.hudson/jobs/lds-30-trunk/builds/2007-06-13_15-02-00/archive/trunk/build/dist/lds.jar
> (Too many open files)
> at java.io.FileOutputStream.open(Native Method)
> at
> java.io.FileOutputStream.<init>(FileOutputStream.java:179)
> at
> java.io.FileOutputStream.<init>(FileOutputStream.java:131)
> at
> org.apache.tools.ant.types.resources.FileResource.getOutputStream(FileResource.java:213)
> at
> org.apache.tools.ant.util.ResourceUtils.copyResource(ResourceUtils.java:379)
> at
> org.apache.tools.ant.util.FileUtils.copyFile(FileUtils.java:508)
> at
> org.apache.tools.ant.util.FileUtils.copyFile(FileUtils.java:323)
> at
> org.apache.tools.ant.taskdefs.Copy.doFileOperations(Copy.java:795)
> ... 11 more
> Recording fingerprints
> Recording test results
> Test reports were found but none of them are new. Did
> tests run?
> Recording Emma reports
> trunk/build/dist/coverage/coverage.xml
> FATAL: failed
> java.io.FileNotFoundException:
> /root/.hudson/jobs/lds-30-trunk/workspace/trunk/build/dist/coverage/coverage.xml
> (Too many open files)
> at java.io.FileInputStream.open(Native Method)
> at
> java.io.FileInputStream.<init>(FileInputStream.java:106)
> at hudson.FilePath$18.invoke(FilePath.java:608)
> at hudson.FilePath$18.invoke(FilePath.java:606)
> at hudson.FilePath.act(FilePath.java:239)
> at hudson.FilePath.copyTo(FilePath.java:606)
> at hudson.FilePath.copyTo(FilePath.java:594)
> at
> hudson.plugins.emma.EmmaPublisher.perform(EmmaPublisher.java:54)
> at hudson.model.Build$RunnerImpl.post(Build.java:137)
> at hudson.model.Run.run(Run.java:572)
> at hudson.model.Build.run(Build.java:101)
> at hudson.model.Executor.run(Executor.java:59)
> finished: FAILURE
>
> [rvanderborght@build ~]$ sudo /usr/sbin/lsof | grep
> hudson
> java      27023          root  mem       REG      
> 8,2    261710     389933
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/xstream-1.1.3.jar
> java      27023          root  mem       REG      
> 8,2     93395     389932
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/xpp3-1.1.3.3.jar
> java      27023          root  mem       REG      
> 8,2      9766     389931
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/wagon-ssh-external-1.0-alpha-6.jar
> java      27023          root  mem       REG      
> 8,2     32265     389930
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/wagon-ssh-1.0-alpha-7.jar
> java      27023          root  mem       REG      
> 8,2     42727     389929
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/wagon-provider-api-1.0-alpha-6.jar
> java      27023          root  mem       REG      
> 8,2      8336     389928
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/wagon-http-lightweight-1.0-alpha-6.jar
> java      27023          root  mem       REG      
> 8,2      4724     389927
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/wagon-file-1.0-alpha-7.jar
> java      27023          root  mem       REG      
> 8,2   1240894     389926
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/svnkit-1.1.2-hudson-2.jar
> java      27023          root  mem       REG      
> 8,2    111296     389925
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/stapler-1.30.jar
> java      27023          root  mem       REG      
> 8,2     64347     389924
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/remoting-1.105.jar
> java      27023          root  mem       REG      
> 8,2    168568     389923
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/plexus-utils-1.1.jar
> java      27023          root  mem       REG      
> 8,2     13407     389922
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/plexus-interactivity-api-1.0-alpha-4.jar
> java      27023          root  mem       REG      
> 8,2    195394     389921
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/plexus-container-default-1.0-alpha-9.jar
> java      27023          root  mem       REG      
> 8,2     44144     389920
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-settings-2.0.4.jar
> java      27023          root  mem       REG      
> 8,2     19819     389919
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-repository-metadata-2.0.4.jar
> java      27023          root  mem       REG      
> 8,2      5000     389918
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-reporting-api-2.0.4.jar
> java      27023          root  mem       REG      
> 8,2    109543     389917
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-project-2.0.4.jar
> java      27023          root  mem       REG      
> 8,2     30421     389916
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-profile-2.0.4.jar
> java      27023          root  mem       REG      
> 8,2     24084     389915
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-plugin-registry-2.0.4.jar
> java      27023          root  mem       REG      
> 8,2     16392     389914
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-plugin-parameter-documenter-2.0.4.jar
> java      27023          root  mem       REG      
> 8,2     32258     389913
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-plugin-descriptor-2.0.4.jar
> java      27023          root  mem       REG      
> 8,2      8341     389912
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-plugin-api-2.0.4.jar
> java      27023          root  mem       REG      
> 8,2      5738     389911
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-monitor-2.0.4.jar
> java      27023          root  mem       REG      
> 8,2     81575     389910
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-model-2.0.4.jar
> java      27023          root  mem       REG      
> 8,2     14545     389909
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-interceptor-1.105.jar
> java      27023          root  mem       REG      
> 8,2      8658     389908
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-error-diagnostics-2.0.4.jar
> java      27023          root  mem       REG      
> 8,2     20615     389907
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-embedder-2.0.4.jar
> java      27023          root  mem       REG      
> 8,2    137173     389906
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-core-2.0.4.jar
> java      27023          root  mem       REG      
> 8,2     49162     389905
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-artifact-manager-2.0.4.jar
> java      27023          root  mem       REG      
> 8,2     80522     389904
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-artifact-2.0.4.jar
> java      27023          root  mem       REG      
> 8,2      6020     389903
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-agent-1.105.jar
> java      27023          root  mem       REG      
> 8,2     72150     389901
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/logkit-1.0.1.jar
> java      27023          root  mem       REG      
> 8,2    358085     389900
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/log4j-1.2.12.jar
> java      27023          root  mem       REG      
> 8,2     16923     389899
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/jstl-1.1.0.jar
> java      27023          root  mem       REG      
> 8,2    117174     389898
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/jsch-0.1.24.jar
> java      27023          root  mem       REG      
> 8,2   1139113     389897
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/jfreechart-1.0.3.jar
> java      27023          root  mem       REG      
> 8,2    238637     389895
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/jaxen-1.1-beta-11.jar
> java      27023          root  mem       REG      
> 8,2   1800974     389893
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/groovy-all-1.0-jsr-06.jar
> java      27023          root  mem       REG      
> 8,2     47997     389892
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/graph-layouter-1.0.jar
> java      27023          root  mem       REG      
> 8,2    244831     389891
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/ganymed-ssh2-build210.jar
> java      27023          root  mem       REG      
> 8,2      5936     389890
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/doxia-sink-api-1.0-alpha-7.jar
> java      27023          root  mem       REG      
> 8,2    313898     389889
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/dom4j-1.6.1.jar
> java      27023          root  mem       REG      
> 8,2     52915     389888
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-logging-1.1.jar
> java      27023          root  mem       REG      
> 8,2    132239     389887
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-jexl-1.1-hudson-20061106.jar
> java      27023          root  mem       REG      
> 8,2     37430     389886
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-jelly-tags-xml-1.1.jar
> java      27023          root  mem       REG      
> 8,2     22479     389885
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-jelly-tags-fmt-1.0.jar
> java      27023          root  mem       REG      
> 8,2     28630     389884
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-jelly-tags-define-1.0.1-hudson-20061106.jar
> java      27023          root  mem       REG      
> 8,2     83613     389882
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-io-1.3.1.jar
> java      27023          root  mem       REG      
> 8,2     31825     389881
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-fileupload-1.1.jar
> java      27023          root  mem       REG      
> 8,2    139966     389880
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-digester-1.7.jar
> java      27023          root  mem       REG      
> 8,2    165119     389879
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-collections-2.1.jar
> java      27023          root  mem       REG      
> 8,2    188671     389878
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-beanutils-1.7.0.jar
> java      27023          root  mem       REG      
> 8,2     37854     389877
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/classworlds-1.1.jar
> java      27023          root  mem       REG      
> 8,2     63966     389876
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/avalon-framework-4.1.3.jar
> java      27023          root  mem       REG      
> 8,2    443432     389875
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/antlr-2.7.6.jar
> java      27023          root  mem       REG      
> 8,2     11734     389874
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/ant-launcher-1.7.0.jar
> java      27023          root  mem       REG      
> 8,2     92792     389873
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/ant-junit-1.7.0.jar
> java      27023          root  mem       REG      
> 8,2   1289806     389872
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/ant-1.7.0.jar
> java      27023          root  mem       REG      
> 8,2    388826     389902
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/mail-1.4.jar
> java      27023          root  mem       REG      
> 8,2    308394     389896
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/jcommon-1.0.6.jar
> java      27023          root  mem       REG      
> 8,2    852231     389894
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/hudson-core-1.105.jar
> java      27023          root  mem       REG      
> 8,2    174753     389883
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-jelly-1.1-hudson-20061106.jar
> java      27023          root  mem       REG      
> 8,2     62983     389871
> /usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/activation-1.1.jar
> java      27023          root 1474w      REG      
> 8,9      4305   11109289
> /home/root/.hudson/jobs/edu-potrero/builds/2007-06-12_12-02-01/changelog.xml
> java      27023          root 1704r      REG      
> 8,9        38   79904958
> /home/root/.hudson/jobs/lds-30-trunk/builds/2007-06-13_15-02-00/revision.txt
> java      27023          root 2011r      REG      
> 8,9        38   79905723
> /home/root/.hudson/jobs/lds-30-trunk/builds/2007-06-13_17-01-56/revision.txt
> java      27023          root 2017w      REG      
> 8,9       347   79906188
> /home/root/.hudson/jobs/lds-30-trunk/builds/2007-06-13_17-01-56/changelog.xml
>
> --- Ray V <[hidden email]> wrote:
>
>> Cool, thanks.  Below is the info I collected from
>> forcing a build (one that failed with this problem)
>> after the last time I saw this happen.  Next time
>> I'll
>> try to get a more detailed log (containing the
>> actual
>> open files).
>>
>> Like I said it's been happening unpredictably over
>> time and I suspect the javadocs might be a cause.
>>
>> Also we're running junit4 unit tests via ant, which
>> took some strange hacks to get working at all (had
>> to
>> set fork="false" for some reason on the junit task).
>>
>> Considering going back to using junit3.  Anyway,
>> here's some minimal info:
>>
>> [root@build ~]# ulimit -a
>> core file size          (blocks, -c) 0
>> data seg size           (kbytes, -d) unlimited
>> file size               (blocks, -f) unlimited
>> pending signals                 (-i) 1024
>> max locked memory       (kbytes, -l) 32
>> max memory size         (kbytes, -m) unlimited
>> open files                      (-n) 1024
>> pipe size            (512 bytes, -p) 8
>> POSIX message queues     (bytes, -q) 819200
>> stack size              (kbytes, -s) 10240
>> cpu time               (seconds, -t) unlimited
>> max user processes              (-u) 73728
>> virtual memory          (kbytes, -v) unlimited
>> file locks                      (-x) unlimited
>>
>> Random sampling of open files while this last build
>> was running:
>> [root@build ~]# lsof | grep java | wc -l 1710
>> [root@build ~]# lsof | grep hudson | wc -l
>> 17
>> [root@build ~]# lsof | grep java | wc -l 1710
>> [root@build ~]# lsof | grep hudson | wc -l
>> 17
>> [root@build ~]# lsof | grep hudson | wc -l
>> 151
>> [root@build ~]# lsof | grep hudson | wc -l
>> 16
>> [root@build ~]# lsof | grep java | wc -l
>> 1844
>> [root@build ~]# lsof | grep hudson | wc -l 150
>> [root@build ~]# lsof | grep java | wc -l
>> 1965
>> [root@build ~]# lsof | grep hudson | wc -l 150
>> [root@build ~]# lsof | grep hudson | wc -l 150
>> [root@build ~]# lsof | grep hudson | wc -l 150
>> [root@build ~]# lsof | grep hudson | wc -l 10
>>
>> --- Kohsuke Kawaguchi <[hidden email]>
>> wrote:
>>
>> > Ray V wrote:
>> > > Thanks for the quick replies.  I tried lsof on
>> the
>> > > hudson process during a build and it got pretty
>> > high.
>> > > The max on the box (redhat) is 1024 I think.  We
>> > may
>> > > bump that up.
>> >
>> > Make sure that you are getting the list from the
>> > hudson process that
>> > runs the webapp, not the child processes that are
>> > doing the build.
>> >
>> > > But you're saying that the hudson proc will
>> start
>> > a
>> > > new process for each build right?  That means
>> that
>> > my
>> > > build isn't causing the problem I guess... could
>> > it be
>> > > the last step where it copies the artifacts?
>> >
>> > Somewhere Hudson must be forgetting to close file
>> > handles. We just need
>> > to find out where, and for that lsof is helpful.
>> >
>> > If you can share with us what kind of files that
>> > Hudson is opening, that
>> > should be a good clue for me to tell where the
>> leak
>> > might be happening.
>> >
>> > If you can't share it with public, please consider
>> > sending it to me
>> > privately.
>> >
>> > > I removed the javadoc documentation temporarily
>> > from
>> > > my config to see if that helps this out.
>> >
>> > Thanks.
>> >
>> > --
>> > Kohsuke Kawaguchi
>> > Sun Microsystems                  
>> > [hidden email]
>> >
>>
>>
>>
>>        
>>
> ____________________________________________________________________________________Be
>> a better Globetrotter. Get better travel answers
>> from someone who knows. Yahoo! Answers - Check it
>> out.
>>
> http://answers.yahoo.com/dir/?link=list&sid=396545469
>>
>>
> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> [hidden email]
>> For additional commands, e-mail:
>> [hidden email]
>>
>>
>
>
>
>  
> ____________________________________________________________________________________
> Finding fabulous fares is fun.  
> Let Yahoo! FareChase search your favorite travel sites to find flight and hotel bargains.
> http://farechase.yahoo.com/promo-generic-14795097
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: too many open files

Ray V-2
Cool, glad I could help squash a bug.  I suspect
there's still a leak somewhere, maybe caused by a
different build from the failing one.  Most builds use
ant, but a couple use hudson's shell scripting
ability.  We had previously doubled the max open files
and it just took longer to see this issue reappear.

The other thing is that lsof shows tons of open pipes
owned by the tomcat process... I think it's the pipes
that are growing; the files seem to be under control.

Anyway, if I get more specific info I'll pass it
along.  Thanks for your help so far.

Another symptom is

--- Kohsuke Kawaguchi <[hidden email]>
wrote:

> Thanks to your report, I was able to fix one place
> where I forgot to
> close. From the list it appears to me that
> changelog.xml file handle for
> write is also leaking, so I take some defensive
> measure to prevent that,
> too.
>
> But looking at your list, of 67 files that are
> listed, 63 are jar files
> in the lib directory, which are obviously not leaks.
> So in any case it
> seems like your need to increase the file handle
> limit.
>
>
> Ray V wrote:
> > This issue came up again... here's the console
> output
> > followed by a list of open files taken after the
> build
> > failed... after waiting a while the open files
> went
> > back down again by itself.  not sure what else to
> try.
> >
> > started
> > Updating http://svn:81/svn/LDS/LDS30/trunk
> > U         build.xml
> > At revision 540
> > [trunk] $ ant build
> > FATAL: command execution failed
> > java.io.IOException: java.io.IOException: Too many
> > open files
> > at
> java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
> > at
> java.lang.ProcessImpl.start(ProcessImpl.java:65)
> > at
> >
>
java.lang.ProcessBuilder.start(ProcessBuilder.java:451)
> > at java.lang.Runtime.exec(Runtime.java:591)
> > at hudson.Proc$LocalProc.<init>(Proc.java:79)
> > at
> >
>
hudson.Launcher$LocalLauncher.launch(Launcher.java:167)

> > at hudson.Launcher.launch(Launcher.java:82)
> > at hudson.Launcher.launch(Launcher.java:70)
> > at hudson.tasks.Ant.perform(Ant.java:113)
> > at
> >
> hudson.model.Build$RunnerImpl.build(Build.java:149)
> > at
> >
> hudson.model.Build$RunnerImpl.doRun(Build.java:121)
> > at
> >
>
hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:121)
> > at hudson.model.Run.run(Run.java:559)
> > at hudson.model.Build.run(Build.java:101)
> > at hudson.model.Executor.run(Executor.java:59)
> > ERROR: Failed to archive artifacts:
> >
>
trunk/build/dist/war/lds.war,trunk/build/dist/lds.jar,trunk/build/dist/lds-dao.jar
> > hudson.util.IOException2: Failed to copy
> >
>
/root/.hudson/jobs/lds-30-trunk/workspace/trunk/build/dist/war/lds.war,trunk/build/dist/lds.jar,trunk/build/dist/lds-dao.jar
> > to
> >
>
/root/.hudson/jobs/lds-30-trunk/builds/2007-06-13_15-02-00/archive
> > at hudson.FilePath$19.invoke(FilePath.java:680)
> > at hudson.FilePath$19.invoke(FilePath.java:647)
> > at hudson.FilePath.act(FilePath.java:239)
> > at
> hudson.FilePath.copyRecursiveTo(FilePath.java:647)
> > at
> >
>
hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:65)
> > at
> hudson.model.Build$RunnerImpl.post(Build.java:137)
> > at hudson.model.Run.run(Run.java:572)
> > at hudson.model.Build.run(Build.java:101)
> > at hudson.model.Executor.run(Executor.java:59)
> > Caused by: Failed to copy
> >
>
/root/.hudson/jobs/lds-30-trunk/workspace/trunk/build/dist/lds.jar
> > to
> >
>
/root/.hudson/jobs/lds-30-trunk/builds/2007-06-13_15-02-00/archive/trunk/build/dist/lds.jar
> > due to java.io.FileNotFoundException
> >
>
/root/.hudson/jobs/lds-30-trunk/builds/2007-06-13_15-02-00/archive/trunk/build/dist/lds.jar
> > (Too many open files)
> > at
> >
>
org.apache.tools.ant.taskdefs.Copy.doFileOperations(Copy.java:807)
> > at
> >
>
hudson.FilePath$19$1CopyImpl.doFileOperations(FilePath.java:661)
> > at
> >
>
org.apache.tools.ant.taskdefs.Copy.execute(Copy.java:534)
> > at hudson.FilePath$19.invoke(FilePath.java:677)
> > ... 8 more
> > Caused by: java.io.FileNotFoundException:
> >
>
/root/.hudson/jobs/lds-30-trunk/builds/2007-06-13_15-02-00/archive/trunk/build/dist/lds.jar
> > (Too many open files)
> > at java.io.FileOutputStream.open(Native Method)
> > at
> >
>
java.io.FileOutputStream.<init>(FileOutputStream.java:179)
> > at
> >
>
java.io.FileOutputStream.<init>(FileOutputStream.java:131)
> > at
> >
>
org.apache.tools.ant.types.resources.FileResource.getOutputStream(FileResource.java:213)
> > at
> >
>
org.apache.tools.ant.util.ResourceUtils.copyResource(ResourceUtils.java:379)
> > at
> >
>
org.apache.tools.ant.util.FileUtils.copyFile(FileUtils.java:508)
> > at
> >
>
org.apache.tools.ant.util.FileUtils.copyFile(FileUtils.java:323)
> > at
> >
>
org.apache.tools.ant.taskdefs.Copy.doFileOperations(Copy.java:795)

> > ... 11 more
> > Recording fingerprints
> > Recording test results
> > Test reports were found but none of them are new.
> Did
> > tests run?
> > Recording Emma reports
> > trunk/build/dist/coverage/coverage.xml
> > FATAL: failed
> > java.io.FileNotFoundException:
> >
>
/root/.hudson/jobs/lds-30-trunk/workspace/trunk/build/dist/coverage/coverage.xml
> > (Too many open files)
> > at java.io.FileInputStream.open(Native Method)
> > at
> >
>
java.io.FileInputStream.<init>(FileInputStream.java:106)
> > at hudson.FilePath$18.invoke(FilePath.java:608)
> > at hudson.FilePath$18.invoke(FilePath.java:606)
> > at hudson.FilePath.act(FilePath.java:239)
> > at hudson.FilePath.copyTo(FilePath.java:606)
> > at hudson.FilePath.copyTo(FilePath.java:594)
> > at
> >
>
hudson.plugins.emma.EmmaPublisher.perform(EmmaPublisher.java:54)

> > at
> hudson.model.Build$RunnerImpl.post(Build.java:137)
> > at hudson.model.Run.run(Run.java:572)
> > at hudson.model.Build.run(Build.java:101)
> > at hudson.model.Executor.run(Executor.java:59)
> > finished: FAILURE
> >
> > [rvanderborght@build ~]$ sudo /usr/sbin/lsof |
> grep
> > hudson
> > java      27023          root  mem       REG      
>
> > 8,2    261710     389933
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/xstream-1.1.3.jar
> > java      27023          root  mem       REG      
>
> > 8,2     93395     389932
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/xpp3-1.1.3.3.jar
> > java      27023          root  mem       REG      
>
> > 8,2      9766     389931
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/wagon-ssh-external-1.0-alpha-6.jar
> > java      27023          root  mem       REG      
>
> > 8,2     32265     389930
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/wagon-ssh-1.0-alpha-7.jar
> > java      27023          root  mem       REG      
>
> > 8,2     42727     389929
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/wagon-provider-api-1.0-alpha-6.jar
> > java      27023          root  mem       REG      
>
> > 8,2      8336     389928
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/wagon-http-lightweight-1.0-alpha-6.jar
> > java      27023          root  mem       REG      
>
> > 8,2      4724     389927
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/wagon-file-1.0-alpha-7.jar
> > java      27023          root  mem       REG      
>
> > 8,2   1240894     389926
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/svnkit-1.1.2-hudson-2.jar
> > java      27023          root  mem       REG      
>
> > 8,2    111296     389925
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/stapler-1.30.jar
> > java      27023          root  mem       REG      
>
> > 8,2     64347     389924
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/remoting-1.105.jar
> > java      27023          root  mem       REG      
>
> > 8,2    168568     389923
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/plexus-utils-1.1.jar
> > java      27023          root  mem       REG      
>
> > 8,2     13407     389922
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/plexus-interactivity-api-1.0-alpha-4.jar
> > java      27023          root  mem       REG      
>
> > 8,2    195394     389921
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/plexus-container-default-1.0-alpha-9.jar
> > java      27023          root  mem       REG      
>
> > 8,2     44144     389920
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-settings-2.0.4.jar
> > java      27023          root  mem       REG      
>
> > 8,2     19819     389919
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-repository-metadata-2.0.4.jar
> > java      27023          root  mem       REG      
>
> > 8,2      5000     389918
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-reporting-api-2.0.4.jar
> > java      27023          root  mem       REG      
>
> > 8,2    109543     389917
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-project-2.0.4.jar
> > java      27023          root  mem       REG      
>
> > 8,2     30421     389916
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-profile-2.0.4.jar
> > java      27023          root  mem       REG      
>
> > 8,2     24084     389915
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-plugin-registry-2.0.4.jar
> > java      27023          root  mem       REG      
>
> > 8,2     16392     389914
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-plugin-parameter-documenter-2.0.4.jar
> > java      27023          root  mem       REG      
>
> > 8,2     32258     389913
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-plugin-descriptor-2.0.4.jar
> > java      27023          root  mem       REG      
>
> > 8,2      8341     389912
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-plugin-api-2.0.4.jar
> > java      27023          root  mem       REG      
>
> > 8,2      5738     389911
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-monitor-2.0.4.jar
> > java      27023          root  mem       REG      
>
> > 8,2     81575     389910
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-model-2.0.4.jar
> > java      27023          root  mem       REG      
>
> > 8,2     14545     389909
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-interceptor-1.105.jar
> > java      27023          root  mem       REG      
>
> > 8,2      8658     389908
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-error-diagnostics-2.0.4.jar
> > java      27023          root  mem       REG      
>
> > 8,2     20615     389907
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-embedder-2.0.4.jar
> > java      27023          root  mem       REG      
>
> > 8,2    137173     389906
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-core-2.0.4.jar
> > java      27023          root  mem       REG      
>
> > 8,2     49162     389905
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-artifact-manager-2.0.4.jar
> > java      27023          root  mem       REG      
>
> > 8,2     80522     389904
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-artifact-2.0.4.jar
> > java      27023          root  mem       REG      
>
> > 8,2      6020     389903
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/maven-agent-1.105.jar
> > java      27023          root  mem       REG      
>
> > 8,2     72150     389901
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/logkit-1.0.1.jar
> > java      27023          root  mem       REG      
>
> > 8,2    358085     389900
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/log4j-1.2.12.jar
> > java      27023          root  mem       REG      
>
> > 8,2     16923     389899
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/jstl-1.1.0.jar
> > java      27023          root  mem       REG      
>
> > 8,2    117174     389898
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/jsch-0.1.24.jar
> > java      27023          root  mem       REG      
>
> > 8,2   1139113     389897
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/jfreechart-1.0.3.jar
> > java      27023          root  mem       REG      
>
> > 8,2    238637     389895
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/jaxen-1.1-beta-11.jar
> > java      27023          root  mem       REG      
>
> > 8,2   1800974     389893
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/groovy-all-1.0-jsr-06.jar
> > java      27023          root  mem       REG      
>
> > 8,2     47997     389892
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/graph-layouter-1.0.jar
> > java      27023          root  mem       REG      
>
> > 8,2    244831     389891
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/ganymed-ssh2-build210.jar
> > java      27023          root  mem       REG      
>
> > 8,2      5936     389890
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/doxia-sink-api-1.0-alpha-7.jar
> > java      27023          root  mem       REG      
>
> > 8,2    313898     389889
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/dom4j-1.6.1.jar
> > java      27023          root  mem       REG      
>
> > 8,2     52915     389888
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-logging-1.1.jar
> > java      27023          root  mem       REG      
>
> > 8,2    132239     389887
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-jexl-1.1-hudson-20061106.jar
> > java      27023          root  mem       REG      
>
> > 8,2     37430     389886
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-jelly-tags-xml-1.1.jar
> > java      27023          root  mem       REG      
>
> > 8,2     22479     389885
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-jelly-tags-fmt-1.0.jar
> > java      27023          root  mem       REG      
>
> > 8,2     28630     389884
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-jelly-tags-define-1.0.1-hudson-20061106.jar
> > java      27023          root  mem       REG      
>
> > 8,2     83613     389882
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-io-1.3.1.jar
> > java      27023          root  mem       REG      
>
> > 8,2     31825     389881
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-fileupload-1.1.jar
> > java      27023          root  mem       REG      
>
> > 8,2    139966     389880
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-digester-1.7.jar
> > java      27023          root  mem       REG      
>
> > 8,2    165119     389879
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-collections-2.1.jar
> > java      27023          root  mem       REG      
>
> > 8,2    188671     389878
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-beanutils-1.7.0.jar
> > java      27023          root  mem       REG      
>
> > 8,2     37854     389877
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/classworlds-1.1.jar
> > java      27023          root  mem       REG      
>
> > 8,2     63966     389876
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/avalon-framework-4.1.3.jar
> > java      27023          root  mem       REG      
>
> > 8,2    443432     389875
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/antlr-2.7.6.jar
> > java      27023          root  mem       REG      
>
> > 8,2     11734     389874
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/ant-launcher-1.7.0.jar
> > java      27023          root  mem       REG      
>
> > 8,2     92792     389873
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/ant-junit-1.7.0.jar
> > java      27023          root  mem       REG      
>
> > 8,2   1289806     389872
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/ant-1.7.0.jar
> > java      27023          root  mem       REG      
>
> > 8,2    388826     389902
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/mail-1.4.jar
> > java      27023          root  mem       REG      
>
> > 8,2    308394     389896
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/jcommon-1.0.6.jar
> > java      27023          root  mem       REG      
>
> > 8,2    852231     389894
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/hudson-core-1.105.jar
> > java      27023          root  mem       REG      
>
> > 8,2    174753     389883
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/commons-jelly-1.1-hudson-20061106.jar
> > java      27023          root  mem       REG      
>
> > 8,2     62983     389871
> >
>
/usr/local/apache-tomcat-5.5.12/webapps/hudson/WEB-INF/lib/activation-1.1.jar
> > java      27023          root 1474w      REG      
>
> > 8,9      4305   11109289
> >
>
/home/root/.hudson/jobs/edu-potrero/builds/2007-06-12_12-02-01/changelog.xml
> > java      27023          root 1704r      REG      
>
> > 8,9        38   79904958
> >
>
/home/root/.hudson/jobs/lds-30-trunk/builds/2007-06-13_15-02-00/revision.txt
> > java      27023          root 2011r      REG      
>
> > 8,9        38   79905723
> >
>
/home/root/.hudson/jobs/lds-30-trunk/builds/2007-06-13_17-01-56/revision.txt
> > java      27023          root 2017w      REG      
>
> > 8,9       347   79906188
> >
>
/home/root/.hudson/jobs/lds-30-trunk/builds/2007-06-13_17-01-56/changelog.xml

> >
> > --- Ray V <[hidden email]> wrote:
> >
> >> Cool, thanks.  Below is the info I collected from
> >> forcing a build (one that failed with this
> problem)
> >> after the last time I saw this happen.  Next time
> >> I'll
> >> try to get a more detailed log (containing the
> >> actual
> >> open files).
> >>
> >> Like I said it's been happening unpredictably
> over
> >> time and I suspect the javadocs might be a cause.
> >>
> >> Also we're running junit4 unit tests via ant,
> which
> >> took some strange hacks to get working at all
> (had
> >> to
> >> set fork="false" for some reason on the junit
> task).
> >>
> >> Considering going back to using junit3.  Anyway,
> >> here's some minimal info:
> >>
> >> [root@build ~]# ulimit -a
> >> core file size          (blocks, -c) 0
> >> data seg size           (kbytes, -d) unlimited
> >> file size               (blocks, -f) unlimited
> >> pending signals                 (-i) 1024
> >> max locked memory       (kbytes, -l) 32
> >> max memory size         (kbytes, -m) unlimited
> >> open files                      (-n) 1024
> >> pipe size            (512 bytes, -p) 8
> >> POSIX message queues     (bytes, -q) 819200
> >> stack size              (kbytes, -s) 10240
> >> cpu time               (seconds, -t) unlimited
> >> max user processes              (-u) 73728
> >> virtual memory          (kbytes, -v) unlimited
> >> file locks                      (-x) unlimited
> >>
> >> Random sampling of open files while this last
> build
> >> was running:
> >> [root@build ~]# lsof | grep java | wc -l 1710
> >> [root@build ~]# lsof | grep hudson | wc -l
> >> 17
> >> [root@build ~]# lsof | grep java | wc -l 1710
> >> [root@build ~]# lsof | grep hudson | wc -l
> >> 17
> >> [root@build ~]# lsof | grep hudson | wc -l
> >> 151
> >> [root@build ~]# lsof | grep hudson | wc -l
> >> 16
> >> [root@build ~]# lsof | grep java | wc -l
> >> 1844
> >> [root@build ~]# lsof | grep hudson | wc -l 150
> >> [root@build ~]# lsof | grep java | wc -l
> >> 1965
> >> [root@build ~]# lsof | grep hudson | wc -l 150
> >> [root@build ~]# lsof | grep hudson | wc -l 150
> >> [root@build ~]# lsof | grep hudson | wc -l 150
> >> [root@build ~]# lsof | grep hudson | wc -l 10
> >>
> >> --- Kohsuke Kawaguchi <[hidden email]>
> >> wrote:
> >>
> >> > Ray V wrote:
> >> > > Thanks for the quick replies.  I tried lsof
> on
> >> the
> >> > > hudson process during a build and it got
> pretty
> >> > high.
> >> > > The max on the box (redhat) is 1024 I think.
> We
> >> > may
> >> > > bump that up.
> >> >
> >> > Make sure that you are getting the list from
> the
> >> > hudson process that
> >> > runs the webapp, not the child processes that
> are
> >> > doing the build.
> >> >
> >> > > But you're saying that the hudson proc will
> >> start
> >> > a
> >> > > new process for each build right?  That means
> >> that
> >> > my
> >> > > build isn't causing the problem I guess...
> could
> >> > it be
> >> > > the last step where it copies the artifacts?
> >> >
> >> > Somewhere Hudson must be forgetting to close
> file
> >> > handles. We just need
> >> > to find out where, and for that lsof is
> helpful.
> >> >
> >> > If you can share with us what kind of files
> that
> >> > Hudson is opening, that
> >> > should be a good clue for me to tell where the
> >> leak
> >> > might be happening.
> >> >
> >> > If you can't share it with public, please
> consider
> >> > sending it to me
> >> > privately.
> >> >
> >> > > I removed the javadoc documentation
> temporarily
> >> > from
> >> > > my config to see if that helps this out.
> >> >
> >> > Thanks.
> >> >
> >> > --
> >> > Kohsuke Kawaguchi
> >> > Sun Microsystems                  
> >> > [hidden email]
> >> >
> >>
> >>
> >>
> >>        
> >>
> >
>
____________________________________________________________________________________Be
> >> a better Globetrotter. Get better travel answers
> >> from someone who knows. Yahoo! Answers - Check it
> >> out.
> >>
> >
>
http://answers.yahoo.com/dir/?link=list&sid=396545469
> >>
> >>
> >
>
---------------------------------------------------------------------

> >> To unsubscribe, e-mail:
> >> [hidden email]
> >> For additional commands, e-mail:
> >> [hidden email]
> >>
> >>
> >
> >
> >
> >  
> >
>
____________________________________________________________________________________
> > Finding fabulous fares is fun.  
> > Let Yahoo! FareChase search your favorite travel
> sites to find flight and hotel bargains.
> > http://farechase.yahoo.com/promo-generic-14795097
> >
> >
>
---------------------------------------------------------------------

> > To unsubscribe, e-mail:
> [hidden email]
> > For additional commands, e-mail:
> [hidden email]
> >
> >
>
>
> --
> Kohsuke Kawaguchi
> Sun Microsystems                  
> [hidden email]
>



       
____________________________________________________________________________________
Sick sense of humor? Visit Yahoo! TV's
Comedy with an Edge to see what's on, when.
http://tv.yahoo.com/collections/222

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: too many open files

Kohsuke Kawaguchi
Administrator
2007/6/14, Ray V <[hidden email]>:
> The other thing is that lsof shows tons of open pipes
> owned by the tomcat process... I think it's the pipes
> that are growing; the files seem to be under control.

Interesting. Is there any way to figure out where those pipes are connected to?

--
Kohsuke Kawaguchi

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: too many open files

Ray V-2
I'm not sure how to get more info on the open pipes
unfortunately.  I found the following post though, and
it has some good descriptions of possible causes of
this kind of thing:

http://www.codecomments.com/archive252-2004-9-288873.html


--- Kohsuke Kawaguchi <[hidden email]> wrote:

> 2007/6/14, Ray V <[hidden email]>:
> > The other thing is that lsof shows tons of open
> pipes
> > owned by the tomcat process... I think it's the
> pipes
> > that are growing; the files seem to be under
> control.
>
> Interesting. Is there any way to figure out where
> those pipes are connected to?
>
> --
> Kohsuke Kawaguchi
>
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> [hidden email]
> For additional commands, e-mail:
> [hidden email]
>
>



       
____________________________________________________________________________________
Building a website is a piece of cake. Yahoo! Small Business gives you all the tools to get online.
http://smallbusiness.yahoo.com/webhosting 

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: too many open files

Ray V-2
Not a totally conclusive test, but I may have narrowed
the open pipes problem down some more.

Below shows the number of open pipes growing after
doing a "build now" on a project that uses "execute
shell" to do its build.  Could be related to not
closing stdin, stdout and stderr streams, but I
haven't yet gone the full mile and looked at the
hudson source.

The build script...
rm -f *.tgz; \
cp -aux trunk ${JOB_NAME}_1-00-${BUILD_NUMBER}_1 && \
tar cvfz ${JOB_NAME}_1-00-${BUILD_NUMBER}_1.tgz
${JOB_NAME}_1-00-${BUILD_NUMBER}_1 --exclude=".svn" &&
\
svn import ${JOB_NAME}_1-00-${BUILD_NUMBER}_1
http://our-site.net:81/svn/our-project/tags/${JOB_NAME}_1-00-${BUILD_NUMBER}_1
-m "adding tag for ${JOB_NAME}_1-00-${BUILD_NUMBER}_1"
&& \
svn import ${JOB_NAME}_1-00-${BUILD_NUMBER}_1.tgz
http://our-site:81/svn/builds/SAT/our-project/${JOB_NAME}_1-00-${BUILD_NUMBER}_1.tgz
-m "adding build for
${JOB_NAME}_1-00-${BUILD_NUMBER}_1"

The pipes (before and after the build)...
[rvanderborght@build ~]$  sudo /usr/sbin/lsof -p 6838
| grep pipe
java    6838 root  309w  FIFO        0,7          
28984266 pipe
java    6838 root  316r  FIFO        0,7          
28984268 pipe
java    6838 root  523w  FIFO        0,7          
29227597 pipe
java    6838 root  524w  FIFO        0,7          
29066615 pipe
java    6838 root  530r  FIFO        0,7          
29227599 pipe
java    6838 root  763w  FIFO        0,7          
29017890 pipe
java    6838 root  768r  FIFO        0,7          
29017892 pipe
java    6838 root  805w  FIFO        0,7          
28999338 pipe
java    6838 root  806w  FIFO        0,7          
29003997 pipe
java    6838 root  807w  FIFO        0,7          
28999350 pipe
java    6838 root  810w  FIFO        0,7          
29004009 pipe
java    6838 root  811w  FIFO        0,7          
29008669 pipe
java    6838 root  812r  FIFO        0,7          
28999340 pipe
java    6838 root  813w  FIFO        0,7          
29013341 pipe
java    6838 root  814r  FIFO        0,7          
28999352 pipe
java    6838 root  818r  FIFO        0,7          
29003999 pipe
java    6838 root  820r  FIFO        0,7          
29004011 pipe
java    6838 root  824r  FIFO        0,7          
29008671 pipe
java    6838 root  827r  FIFO        0,7          
29013343 pipe
java    6838 root  899w  FIFO        0,7          
28994690 pipe
java    6838 root  901w  FIFO        0,7          
28989478 pipe
java    6838 root  908r  FIFO        0,7          
28989480 pipe
java    6838 root  909r  FIFO        0,7          
28994692 pipe
[rvanderborght@build ~]$  sudo /usr/sbin/lsof -p 6838
| grep pipe | wc -l
23

[rvanderborght@build ~]$  sudo /usr/sbin/lsof -p 6838
| grep pipe
java    6838 root   36w  FIFO        0,7          
29244960 pipe
java    6838 root   37w  FIFO        0,7          
29244972 pipe
java    6838 root   38w  FIFO        0,7          
29244984 pipe
java    6838 root   39r  FIFO        0,7          
29244962 pipe
java    6838 root   40w  FIFO        0,7          
29244996 pipe
java    6838 root   41r  FIFO        0,7          
29244974 pipe
java    6838 root   42w  FIFO        0,7          
29245008 pipe
java    6838 root   43r  FIFO        0,7          
29244986 pipe
java    6838 root   44w  FIFO        0,7          
29245020 pipe
java    6838 root   45r  FIFO        0,7          
29244998 pipe
java    6838 root   47r  FIFO        0,7          
29245010 pipe
java    6838 root   48w  FIFO        0,7          
29245040 pipe
java    6838 root   49r  FIFO        0,7          
29245022 pipe
java    6838 root   52r  FIFO        0,7          
29245042 pipe
java    6838 root  309w  FIFO        0,7          
28984266 pipe
java    6838 root  316r  FIFO        0,7          
28984268 pipe
java    6838 root  523w  FIFO        0,7          
29227597 pipe
java    6838 root  524w  FIFO        0,7          
29066615 pipe
java    6838 root  530r  FIFO        0,7          
29227599 pipe
java    6838 root  763w  FIFO        0,7          
29017890 pipe
java    6838 root  768r  FIFO        0,7          
29017892 pipe
java    6838 root  805w  FIFO        0,7          
28999338 pipe
java    6838 root  806w  FIFO        0,7          
29003997 pipe
java    6838 root  807w  FIFO        0,7          
28999350 pipe
java    6838 root  810w  FIFO        0,7          
29004009 pipe
java    6838 root  811w  FIFO        0,7          
29008669 pipe
java    6838 root  812r  FIFO        0,7          
28999340 pipe
java    6838 root  813w  FIFO        0,7          
29013341 pipe
java    6838 root  814r  FIFO        0,7          
28999352 pipe
java    6838 root  818r  FIFO        0,7          
29003999 pipe
java    6838 root  820r  FIFO        0,7          
29004011 pipe
java    6838 root  824r  FIFO        0,7          
29008671 pipe
java    6838 root  827r  FIFO        0,7          
29013343 pipe
java    6838 root  899w  FIFO        0,7          
28994690 pipe
java    6838 root  901w  FIFO        0,7          
28989478 pipe
java    6838 root  908r  FIFO        0,7          
28989480 pipe
java    6838 root  909r  FIFO        0,7          
28994692 pipe
[rvanderborght@build ~]$  sudo /usr/sbin/lsof -p 6838
| grep pipe | wc -l
37

--- Ray V <[hidden email]> wrote:

> I'm not sure how to get more info on the open pipes
> unfortunately.  I found the following post though,
> and
> it has some good descriptions of possible causes of
> this kind of thing:
>
>
http://www.codecomments.com/archive252-2004-9-288873.html

>
>
> --- Kohsuke Kawaguchi <[hidden email]> wrote:
>
> > 2007/6/14, Ray V <[hidden email]>:
> > > The other thing is that lsof shows tons of open
> > pipes
> > > owned by the tomcat process... I think it's the
> > pipes
> > > that are growing; the files seem to be under
> > control.
> >
> > Interesting. Is there any way to figure out where
> > those pipes are connected to?
> >
> > --
> > Kohsuke Kawaguchi
> >
> >
>
---------------------------------------------------------------------

> > To unsubscribe, e-mail:
> > [hidden email]
> > For additional commands, e-mail:
> > [hidden email]
> >
> >
>
>
>
>        
>
____________________________________________________________________________________
> Building a website is a piece of cake. Yahoo! Small
> Business gives you all the tools to get online.
> http://smallbusiness.yahoo.com/webhosting 
>
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> [hidden email]
> For additional commands, e-mail:
> [hidden email]
>
>

--- Ray V <[hidden email]> wrote:

> I'm not sure how to get more info on the open pipes
> unfortunately.  I found the following post though,
> and
> it has some good descriptions of possible causes of
> this kind of thing:
>
>
http://www.codecomments.com/archive252-2004-9-288873.html

>
>
> --- Kohsuke Kawaguchi <[hidden email]> wrote:
>
> > 2007/6/14, Ray V <[hidden email]>:
> > > The other thing is that lsof shows tons of open
> > pipes
> > > owned by the tomcat process... I think it's the
> > pipes
> > > that are growing; the files seem to be under
> > control.
> >
> > Interesting. Is there any way to figure out where
> > those pipes are connected to?
> >
> > --
> > Kohsuke Kawaguchi
> >
> >
>
---------------------------------------------------------------------

> > To unsubscribe, e-mail:
> > [hidden email]
> > For additional commands, e-mail:
> > [hidden email]
> >
> >
>
>
>
>        
>
____________________________________________________________________________________
> Building a website is a piece of cake. Yahoo! Small
> Business gives you all the tools to get online.
> http://smallbusiness.yahoo.com/webhosting 
>
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> [hidden email]
> For additional commands, e-mail:
> [hidden email]
>
>



      ____________________________________________________________________________________
Fussy? Opinionated? Impossible to please? Perfect.  Join Yahoo!'s user panel and lay it on us. http://surveylink.yahoo.com/gmrs/yahoo_panel_invite.asp?a=7 

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: too many open files

Tomasz Sterna-2
In reply to this post by Ray V-2
Dnia 30-05-2007, śro o godzinie 16:28 -0700, Ray V napisał(a):
> FATAL: command execution failed
> java.io.IOException: java.io.IOException: Too many
> open files

This looks like the problem I reported earlier.
In my case these were dangling sockets.

I resorted to nightly Tomcat restarts to overcome the problem.


--
Tomasz Sterna
eo Networks Sp. z o.o.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: too many open files

Kohsuke Kawaguchi
Administrator
In reply to this post by Ray V-2
Ray V wrote:
> Not a totally conclusive test, but I may have narrowed
> the open pipes problem down some more.
>
> Below shows the number of open pipes growing after
> doing a "build now" on a project that uses "execute
> shell" to do its build.  Could be related to not
> closing stdin, stdout and stderr streams, but I
> haven't yet gone the full mile and looked at the
> hudson source.

I checked the code quickly and it does seem to close the stream. I
modified the code a bit so that it closes the stream even in case of an
error. So let's see if that makes any difference in 1.114.

One more question. If you hit the URL "http://server/hudson/gc", it will
  trigger System.gc(). Could that be used as a workaround to keep the
open pipes low?

>
> The build script...
> rm -f *.tgz; \
> cp -aux trunk ${JOB_NAME}_1-00-${BUILD_NUMBER}_1 && \
> tar cvfz ${JOB_NAME}_1-00-${BUILD_NUMBER}_1.tgz
> ${JOB_NAME}_1-00-${BUILD_NUMBER}_1 --exclude=".svn" &&
> \
> svn import ${JOB_NAME}_1-00-${BUILD_NUMBER}_1
> http://our-site.net:81/svn/our-project/tags/${JOB_NAME}_1-00-${BUILD_NUMBER}_1
> -m "adding tag for ${JOB_NAME}_1-00-${BUILD_NUMBER}_1"
> && \
> svn import ${JOB_NAME}_1-00-${BUILD_NUMBER}_1.tgz
> http://our-site:81/svn/builds/SAT/our-project/${JOB_NAME}_1-00-${BUILD_NUMBER}_1.tgz
> -m "adding build for
> ${JOB_NAME}_1-00-${BUILD_NUMBER}_1"
>
> The pipes (before and after the build)...
> [rvanderborght@build ~]$  sudo /usr/sbin/lsof -p 6838
> | grep pipe
> java    6838 root  309w  FIFO        0,7          
> 28984266 pipe
> java    6838 root  316r  FIFO        0,7          
> 28984268 pipe
> java    6838 root  523w  FIFO        0,7          
> 29227597 pipe
> java    6838 root  524w  FIFO        0,7          
> 29066615 pipe
> java    6838 root  530r  FIFO        0,7          
> 29227599 pipe
> java    6838 root  763w  FIFO        0,7          
> 29017890 pipe
> java    6838 root  768r  FIFO        0,7          
> 29017892 pipe
> java    6838 root  805w  FIFO        0,7          
> 28999338 pipe
> java    6838 root  806w  FIFO        0,7          
> 29003997 pipe
> java    6838 root  807w  FIFO        0,7          
> 28999350 pipe
> java    6838 root  810w  FIFO        0,7          
> 29004009 pipe
> java    6838 root  811w  FIFO        0,7          
> 29008669 pipe
> java    6838 root  812r  FIFO        0,7          
> 28999340 pipe
> java    6838 root  813w  FIFO        0,7          
> 29013341 pipe
> java    6838 root  814r  FIFO        0,7          
> 28999352 pipe
> java    6838 root  818r  FIFO        0,7          
> 29003999 pipe
> java    6838 root  820r  FIFO        0,7          
> 29004011 pipe
> java    6838 root  824r  FIFO        0,7          
> 29008671 pipe
> java    6838 root  827r  FIFO        0,7          
> 29013343 pipe
> java    6838 root  899w  FIFO        0,7          
> 28994690 pipe
> java    6838 root  901w  FIFO        0,7          
> 28989478 pipe
> java    6838 root  908r  FIFO        0,7          
> 28989480 pipe
> java    6838 root  909r  FIFO        0,7          
> 28994692 pipe
> [rvanderborght@build ~]$  sudo /usr/sbin/lsof -p 6838
> | grep pipe | wc -l
> 23
>
> [rvanderborght@build ~]$  sudo /usr/sbin/lsof -p 6838
> | grep pipe
> java    6838 root   36w  FIFO        0,7          
> 29244960 pipe
> java    6838 root   37w  FIFO        0,7          
> 29244972 pipe
> java    6838 root   38w  FIFO        0,7          
> 29244984 pipe
> java    6838 root   39r  FIFO        0,7          
> 29244962 pipe
> java    6838 root   40w  FIFO        0,7          
> 29244996 pipe
> java    6838 root   41r  FIFO        0,7          
> 29244974 pipe
> java    6838 root   42w  FIFO        0,7          
> 29245008 pipe
> java    6838 root   43r  FIFO        0,7          
> 29244986 pipe
> java    6838 root   44w  FIFO        0,7          
> 29245020 pipe
> java    6838 root   45r  FIFO        0,7          
> 29244998 pipe
> java    6838 root   47r  FIFO        0,7          
> 29245010 pipe
> java    6838 root   48w  FIFO        0,7          
> 29245040 pipe
> java    6838 root   49r  FIFO        0,7          
> 29245022 pipe
> java    6838 root   52r  FIFO        0,7          
> 29245042 pipe
> java    6838 root  309w  FIFO        0,7          
> 28984266 pipe
> java    6838 root  316r  FIFO        0,7          
> 28984268 pipe
> java    6838 root  523w  FIFO        0,7          
> 29227597 pipe
> java    6838 root  524w  FIFO        0,7          
> 29066615 pipe
> java    6838 root  530r  FIFO        0,7          
> 29227599 pipe
> java    6838 root  763w  FIFO        0,7          
> 29017890 pipe
> java    6838 root  768r  FIFO        0,7          
> 29017892 pipe
> java    6838 root  805w  FIFO        0,7          
> 28999338 pipe
> java    6838 root  806w  FIFO        0,7          
> 29003997 pipe
> java    6838 root  807w  FIFO        0,7          
> 28999350 pipe
> java    6838 root  810w  FIFO        0,7          
> 29004009 pipe
> java    6838 root  811w  FIFO        0,7          
> 29008669 pipe
> java    6838 root  812r  FIFO        0,7          
> 28999340 pipe
> java    6838 root  813w  FIFO        0,7          
> 29013341 pipe
> java    6838 root  814r  FIFO        0,7          
> 28999352 pipe
> java    6838 root  818r  FIFO        0,7          
> 29003999 pipe
> java    6838 root  820r  FIFO        0,7          
> 29004011 pipe
> java    6838 root  824r  FIFO        0,7          
> 29008671 pipe
> java    6838 root  827r  FIFO        0,7          
> 29013343 pipe
> java    6838 root  899w  FIFO        0,7          
> 28994690 pipe
> java    6838 root  901w  FIFO        0,7          
> 28989478 pipe
> java    6838 root  908r  FIFO        0,7          
> 28989480 pipe
> java    6838 root  909r  FIFO        0,7          
> 28994692 pipe
> [rvanderborght@build ~]$  sudo /usr/sbin/lsof -p 6838
> | grep pipe | wc -l
> 37
>
> --- Ray V <[hidden email]> wrote:
>
>> I'm not sure how to get more info on the open pipes
>> unfortunately.  I found the following post though,
>> and
>> it has some good descriptions of possible causes of
>> this kind of thing:
>>
>>
> http://www.codecomments.com/archive252-2004-9-288873.html
>>
>>
>> --- Kohsuke Kawaguchi <[hidden email]> wrote:
>>
>> > 2007/6/14, Ray V <[hidden email]>:
>> > > The other thing is that lsof shows tons of open
>> > pipes
>> > > owned by the tomcat process... I think it's the
>> > pipes
>> > > that are growing; the files seem to be under
>> > control.
>> >
>> > Interesting. Is there any way to figure out where
>> > those pipes are connected to?
>> >
>> > --
>> > Kohsuke Kawaguchi
>> >
>> >
>>
> ---------------------------------------------------------------------
>> > To unsubscribe, e-mail:
>> > [hidden email]
>> > For additional commands, e-mail:
>> > [hidden email]
>> >
>> >
>>
>>
>>
>>        
>>
> ____________________________________________________________________________________
>> Building a website is a piece of cake. Yahoo! Small
>> Business gives you all the tools to get online.
>> http://smallbusiness.yahoo.com/webhosting 
>>
>>
> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> [hidden email]
>> For additional commands, e-mail:
>> [hidden email]
>>
>>
>
> --- Ray V <[hidden email]> wrote:
>
>> I'm not sure how to get more info on the open pipes
>> unfortunately.  I found the following post though,
>> and
>> it has some good descriptions of possible causes of
>> this kind of thing:
>>
>>
> http://www.codecomments.com/archive252-2004-9-288873.html
>>
>>
>> --- Kohsuke Kawaguchi <[hidden email]> wrote:
>>
>> > 2007/6/14, Ray V <[hidden email]>:
>> > > The other thing is that lsof shows tons of open
>> > pipes
>> > > owned by the tomcat process... I think it's the
>> > pipes
>> > > that are growing; the files seem to be under
>> > control.
>> >
>> > Interesting. Is there any way to figure out where
>> > those pipes are connected to?
>> >
>> > --
>> > Kohsuke Kawaguchi
>> >
>> >
>>
> ---------------------------------------------------------------------
>> > To unsubscribe, e-mail:
>> > [hidden email]
>> > For additional commands, e-mail:
>> > [hidden email]
>> >
>> >
>>
>>
>>
>>        
>>
> ____________________________________________________________________________________
>> Building a website is a piece of cake. Yahoo! Small
>> Business gives you all the tools to get online.
>> http://smallbusiness.yahoo.com/webhosting 
>>
>>
> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> [hidden email]
>> For additional commands, e-mail:
>> [hidden email]
>>
>>
>
>
>
>       ____________________________________________________________________________________
> Fussy? Opinionated? Impossible to please? Perfect.  Join Yahoo!'s user panel and lay it on us. http://surveylink.yahoo.com/gmrs/yahoo_panel_invite.asp?a=7 
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: too many open files

Ray V-2
I tried the gc link and verified that it closed a
bunch of open pipes for me.  This workaround should
come in really handy.

If I get the chance I'll dig into the source and try
and debug the underlying problem, but in the meantime
this workaround looks like it fits the bill nicely.
Thanks a bunch!

--- Kohsuke Kawaguchi <[hidden email]>
wrote:

> Ray V wrote:
> > Not a totally conclusive test, but I may have
> narrowed
> > the open pipes problem down some more.
> >
> > Below shows the number of open pipes growing after
> > doing a "build now" on a project that uses
> "execute
> > shell" to do its build.  Could be related to not
> > closing stdin, stdout and stderr streams, but I
> > haven't yet gone the full mile and looked at the
> > hudson source.
>
> I checked the code quickly and it does seem to close
> the stream. I
> modified the code a bit so that it closes the stream
> even in case of an
> error. So let's see if that makes any difference in
> 1.114.
>
> One more question. If you hit the URL
> "http://server/hudson/gc", it will
>   trigger System.gc(). Could that be used as a
> workaround to keep the
> open pipes low?
>
> >
> > The build script...
> > rm -f *.tgz; \
> > cp -aux trunk ${JOB_NAME}_1-00-${BUILD_NUMBER}_1
> && \
> > tar cvfz ${JOB_NAME}_1-00-${BUILD_NUMBER}_1.tgz
> > ${JOB_NAME}_1-00-${BUILD_NUMBER}_1
> --exclude=".svn" &&
> > \
> > svn import ${JOB_NAME}_1-00-${BUILD_NUMBER}_1
> >
>
http://our-site.net:81/svn/our-project/tags/${JOB_NAME}_1-00-${BUILD_NUMBER}_1
> > -m "adding tag for
> ${JOB_NAME}_1-00-${BUILD_NUMBER}_1"
> > && \
> > svn import ${JOB_NAME}_1-00-${BUILD_NUMBER}_1.tgz
> >
>
http://our-site:81/svn/builds/SAT/our-project/${JOB_NAME}_1-00-${BUILD_NUMBER}_1.tgz

> > -m "adding build for
> > ${JOB_NAME}_1-00-${BUILD_NUMBER}_1"
> >
> > The pipes (before and after the build)...
> > [rvanderborght@build ~]$  sudo /usr/sbin/lsof -p
> 6838
> > | grep pipe
> > java    6838 root  309w  FIFO        0,7          
> > 28984266 pipe
> > java    6838 root  316r  FIFO        0,7          
> > 28984268 pipe
> > java    6838 root  523w  FIFO        0,7          
> > 29227597 pipe
> > java    6838 root  524w  FIFO        0,7          
> > 29066615 pipe
> > java    6838 root  530r  FIFO        0,7          
> > 29227599 pipe
> > java    6838 root  763w  FIFO        0,7          
> > 29017890 pipe
> > java    6838 root  768r  FIFO        0,7          
> > 29017892 pipe
> > java    6838 root  805w  FIFO        0,7          
> > 28999338 pipe
> > java    6838 root  806w  FIFO        0,7          
> > 29003997 pipe
> > java    6838 root  807w  FIFO        0,7          
> > 28999350 pipe
> > java    6838 root  810w  FIFO        0,7          
> > 29004009 pipe
> > java    6838 root  811w  FIFO        0,7          
> > 29008669 pipe
> > java    6838 root  812r  FIFO        0,7          
> > 28999340 pipe
> > java    6838 root  813w  FIFO        0,7          
> > 29013341 pipe
> > java    6838 root  814r  FIFO        0,7          
> > 28999352 pipe
> > java    6838 root  818r  FIFO        0,7          
> > 29003999 pipe
> > java    6838 root  820r  FIFO        0,7          
> > 29004011 pipe
> > java    6838 root  824r  FIFO        0,7          
> > 29008671 pipe
> > java    6838 root  827r  FIFO        0,7          
> > 29013343 pipe
> > java    6838 root  899w  FIFO        0,7          
> > 28994690 pipe
> > java    6838 root  901w  FIFO        0,7          
> > 28989478 pipe
> > java    6838 root  908r  FIFO        0,7          
> > 28989480 pipe
> > java    6838 root  909r  FIFO        0,7          
> > 28994692 pipe
> > [rvanderborght@build ~]$  sudo /usr/sbin/lsof -p
> 6838
> > | grep pipe | wc -l
> > 23
> >
> > [rvanderborght@build ~]$  sudo /usr/sbin/lsof -p
> 6838
> > | grep pipe
> > java    6838 root   36w  FIFO        0,7          
> > 29244960 pipe
> > java    6838 root   37w  FIFO        0,7          
> > 29244972 pipe
> > java    6838 root   38w  FIFO        0,7          
> > 29244984 pipe
> > java    6838 root   39r  FIFO        0,7          
> > 29244962 pipe
> > java    6838 root   40w  FIFO        0,7          
> > 29244996 pipe
> > java    6838 root   41r  FIFO        0,7          
> > 29244974 pipe
> > java    6838 root   42w  FIFO        0,7          
> > 29245008 pipe
> > java    6838 root   43r  FIFO        0,7          
> > 29244986 pipe
> > java    6838 root   44w  FIFO        0,7          
> > 29245020 pipe
> > java    6838 root   45r  FIFO        0,7          
> > 29244998 pipe
> > java    6838 root   47r  FIFO        0,7          
> > 29245010 pipe
> > java    6838 root   48w  FIFO        0,7          
> > 29245040 pipe
> > java    6838 root   49r  FIFO        0,7          
> > 29245022 pipe
> > java    6838 root   52r  FIFO        0,7          
> > 29245042 pipe
> > java    6838 root  309w  FIFO        0,7          
> > 28984266 pipe
> > java    6838 root  316r  FIFO        0,7          
> > 28984268 pipe
> > java    6838 root  523w  FIFO        0,7          
> > 29227597 pipe
> > java    6838 root  524w  FIFO        0,7          
> > 29066615 pipe
> > java    6838 root  530r  FIFO        0,7          
> > 29227599 pipe
> > java    6838 root  763w  FIFO        0,7          
> > 29017890 pipe
> > java    6838 root  768r  FIFO        0,7          
> > 29017892 pipe
> > java    6838 root  805w  FIFO        0,7          
> > 28999338 pipe
> > java    6838 root  806w  FIFO        0,7          
> > 29003997 pipe
> > java    6838 root  807w  FIFO        0,7          
> > 28999350 pipe
> > java    6838 root  810w  FIFO        0,7          
> > 29004009 pipe
> > java    6838 root  811w  FIFO        0,7          
> > 29008669 pipe
> > java    6838 root  812r  FIFO        0,7          
> > 28999340 pipe
> > java    6838 root  813w  FIFO        0,7          
> > 29013341 pipe
> > java    6838 root  814r  FIFO        0,7          
> > 28999352 pipe
> > java    6838 root  818r  FIFO        0,7          
> > 29003999 pipe
> > java    6838 root  820r  FIFO        0,7          
> > 29004011 pipe
> > java    6838 root  824r  FIFO        0,7          
> > 29008671 pipe
> > java    6838 root  827r  FIFO        0,7          
> > 29013343 pipe
> > java    6838 root  899w  FIFO        0,7          
> > 28994690 pipe
> > java    6838 root  901w  FIFO        0,7          
> > 28989478 pipe
> > java    6838 root  908r  FIFO        0,7          
> > 28989480 pipe
> > java    6838 root  909r  FIFO        0,7          
> > 28994692 pipe
> > [rvanderborght@build ~]$  sudo /usr/sbin/lsof -p
> 6838
> > | grep pipe | wc -l
> > 37
> >
> > --- Ray V <[hidden email]> wrote:
> >
> >> I'm not sure how to get more info on the open
> pipes
> >> unfortunately.  I found the following post
> though,
> >> and
> >> it has some good descriptions of possible causes
> of
> >> this kind of thing:
> >>
> >>
> >
>
http://www.codecomments.com/archive252-2004-9-288873.html

> >>
> >>
> >> --- Kohsuke Kawaguchi <[hidden email]> wrote:
> >>
> >> > 2007/6/14, Ray V <[hidden email]>:
> >> > > The other thing is that lsof shows tons of
> open
> >> > pipes
> >> > > owned by the tomcat process... I think it's
> the
> >> > pipes
> >> > > that are growing; the files seem to be under
> >> > control.
> >> >
> >> > Interesting. Is there any way to figure out
> where
> >> > those pipes are connected to?
> >> >
> >> > --
> >> > Kohsuke Kawaguchi
> >> >
> >> >
> >>
> >
>
---------------------------------------------------------------------

> >> > To unsubscribe, e-mail:
> >> > [hidden email]
> >> > For additional commands, e-mail:
> >> > [hidden email]
> >> >
> >> >
> >>
> >>
> >>
> >>        
> >>
> >
>
____________________________________________________________________________________
> >> Building a website is a piece of cake. Yahoo!
> Small
> >> Business gives you all the tools to get online.
> >> http://smallbusiness.yahoo.com/webhosting 
> >>
> >>
> >
>
---------------------------------------------------------------------

> >> To unsubscribe, e-mail:
> >> [hidden email]
> >> For additional commands, e-mail:
> >> [hidden email]
> >>
> >>
> >
> > --- Ray V <[hidden email]> wrote:
> >
> >> I'm not sure how to get more info on the open
> pipes
> >> unfortunately.  I found the following post
> though,
> >> and
> >> it has some good descriptions of possible causes
> of
> >> this kind of thing:
> >>
> >>
> >
>
http://www.codecomments.com/archive252-2004-9-288873.html

> >>
> >>
> >> --- Kohsuke Kawaguchi <[hidden email]> wrote:
> >>
> >> > 2007/6/14, Ray V <[hidden email]>:
> >> > > The other thing is that lsof shows tons of
> open
> >> > pipes
> >> > > owned by the tomcat process... I think it's
> the
> >> > pipes
> >> > > that are growing; the files seem to be under
> >> > control.
> >> >
> >> > Interesting. Is there any way to figure out
> where
> >> > those pipes are connected to?
> >> >
> >> > --
> >> > Kohsuke Kawaguchi
> >> >
> >> >
> >>
> >
>
---------------------------------------------------------------------

> >> > To unsubscribe, e-mail:
> >> > [hidden email]
> >> > For additional commands, e-mail:
> >> > [hidden email]
> >> >
> >> >
> >>
> >>
> >>
> >>        
> >>
> >
>
____________________________________________________________________________________
> >> Building a website is a piece of cake. Yahoo!
> Small
> >> Business gives you all the tools to get online.
> >> http://smallbusiness.yahoo.com/webhosting 
> >>
> >>
> >
>
---------------------------------------------------------------------

> >> To unsubscribe, e-mail:
> >> [hidden email]
> >> For additional commands, e-mail:
> >> [hidden email]
> >>
> >>
> >
> >
> >
> >      
>
____________________________________________________________________________________
> > Fussy? Opinionated? Impossible to please? Perfect.
>  Join Yahoo!'s user panel and lay it on us.
>
http://surveylink.yahoo.com/gmrs/yahoo_panel_invite.asp?a=7
>
> >
> >
>
---------------------------------------------------------------------

> > To unsubscribe, e-mail:
> [hidden email]
> > For additional commands, e-mail:
> [hidden email]
> >
> >
>
>
> --
> Kohsuke Kawaguchi
> Sun Microsystems                  
> [hidden email]
>



       
____________________________________________________________________________________
Yahoo! oneSearch: Finally, mobile search
that gives answers, not web links.
http://mobile.yahoo.com/mobileweb/onesearch?refer=1ONXIC

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: too many open files

Kohsuke Kawaguchi
Administrator
Ray V wrote:
> I tried the gc link and verified that it closed a
> bunch of open pipes for me.  This workaround should
> come in really handy.
>
> If I get the chance I'll dig into the source and try
> and debug the underlying problem, but in the meantime
> this workaround looks like it fits the bill nicely.
> Thanks a bunch!

Thanks. That would be great.

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment