spawning processes with Ant

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

spawning processes with Ant

Tom Huybrechts
Hi all,

there were a few threads in the previous months about builds not
terminating when spawning a process from Ant. I am facing the same
issue. I have a build that does nothing else then download an
application server and start it. I use ant exec+spawn to do this, but
I need to find away to really detach stdout+stderr of the application
server from the original build.

Does anybody know a workaround for this ?

Tom

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: spawning processes with Ant

Kohsuke Kawaguchi
Administrator
Tom Huybrechts wrote:

> Hi all,
>
> there were a few threads in the previous months about builds not
> terminating when spawning a process from Ant. I am facing the same
> issue. I have a build that does nothing else then download an
> application server and start it. I use ant exec+spawn to do this, but
> I need to find away to really detach stdout+stderr of the application
> server from the original build.
>
> Does anybody know a workaround for this ?
Not too sure, but wasn't the 'at' command supposed to do something like
that?

More importantly, if you launch a daemon process from a build, how do
you make sure that they are properly terminated? Won't you end up with
too many processes?

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: spawning processes with Ant

mitch gottlieb-2
In reply to this post by Tom Huybrechts
I actually had that same issue!

Using spawn/forks in ant to do any other process during the build would
freeze the Hudson thread. I thought it might be what the ant call was
spawning but that didn't seem to matter.

My workaround was to set a scheduled task to start the service the same time
everyday instead of letting the process run after a Hudson build. so I
basically circumvented Hudson.

To solve the issue in Hudson is still open to ideas.

-----Original Message-----
From: Tom Huybrechts [mailto:[hidden email]]
Sent: Tuesday, July 03, 2007 7:00 AM
To: [hidden email]
Subject: spawning processes with Ant

Hi all,

there were a few threads in the previous months about builds not
terminating when spawning a process from Ant. I am facing the same
issue. I have a build that does nothing else then download an
application server and start it. I use ant exec+spawn to do this, but
I need to find away to really detach stdout+stderr of the application
server from the original build.

Does anybody know a workaround for this ?

Tom

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: spawning processes with Ant

Tom Huybrechts
In reply to this post by Kohsuke Kawaguchi
The first part of starting the new server, is shutting down any
running instances, so the number of processes is not a problem.

If I understand correctly, the problem is that Hudson waits for the
stdout and stderr handles to close. Would it be possible to make this
configurable, and have Hudson stop the build when the process ends ?

Tom

On 7/3/07, Kohsuke Kawaguchi <[hidden email]> wrote:

> Tom Huybrechts wrote:
> > Hi all,
> >
> > there were a few threads in the previous months about builds not
> > terminating when spawning a process from Ant. I am facing the same
> > issue. I have a build that does nothing else then download an
> > application server and start it. I use ant exec+spawn to do this, but
> > I need to find away to really detach stdout+stderr of the application
> > server from the original build.
> >
> > Does anybody know a workaround for this ?
>
> Not too sure, but wasn't the 'at' command supposed to do something like
> that?
>
> More importantly, if you launch a daemon process from a build, how do
> you make sure that they are properly terminated? Won't you end up with
> too many processes?
>
> --
> Kohsuke Kawaguchi
> Sun Microsystems                   [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: spawning processes with Ant

Kohsuke Kawaguchi
Administrator
Tom Huybrechts wrote:
> The first part of starting the new server, is shutting down any
> running instances, so the number of processes is not a problem.
>
> If I understand correctly, the problem is that Hudson waits for the
> stdout and stderr handles to close. Would it be possible to make this
> configurable, and have Hudson stop the build when the process ends ?

I guess it's possible. The problem is that Hudson cannot reliably
capture all the output from the process if we do that.

It's possible that the process writes a lot of stuff (which gets
buffered in the pipe), then quit. If we consider the build over as soon
as the process terminates, then we might not be able to drain the buffer.


I think it's important to point out that the real problem is in those
programs that get spawned. If they take proper measures to be a real
daemon, this problem won't happen. Hudson blocks forever because
stdout/stderr pipes between Hudson and the child process are still open,
and that's because the daemon process inherits them and forgot to close
them.

It looks like there are tools that you can use to work around this
issue: http://www.clapper.org/software/daemonize/

We should at least put this on Wiki.

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: spawning processes with Ant

Tom Huybrechts
On 7/4/07, Kohsuke Kawaguchi <[hidden email]> wrote:

> Tom Huybrechts wrote:
> > The first part of starting the new server, is shutting down any
> > running instances, so the number of processes is not a problem.
> >
> > If I understand correctly, the problem is that Hudson waits for the
> > stdout and stderr handles to close. Would it be possible to make this
> > configurable, and have Hudson stop the build when the process ends ?
>
> I guess it's possible. The problem is that Hudson cannot reliably
> capture all the output from the process if we do that.
>
> It's possible that the process writes a lot of stuff (which gets
> buffered in the pipe), then quit. If we consider the build over as soon
> as the process terminates, then we might not be able to drain the buffer.
>
>
> I think it's important to point out that the real problem is in those
> programs that get spawned. If they take proper measures to be a real
> daemon, this problem won't happen. Hudson blocks forever because
> stdout/stderr pipes between Hudson and the child process are still open,
> and that's because the daemon process inherits them and forgot to close
> them.
>
> It looks like there are tools that you can use to work around this
> issue: http://www.clapper.org/software/daemonize/
>
> We should at least put this on Wiki.
>

Unfortunately I'm on Windows, so that won't work for me. There are
tools that let your app run like a Windows service (srvany), but
that's a bit overkill. I thought nohup.exe would do the trick, but
that doesn't work either.

Wouldn't this be a workable solution ?
- provide an option to ignore child processes,
- wait for the process to finish,
- and then wait for the buffers to be drained,

If you ignore child processes, then nothing can be added to the
buffers after the process ends, so once it is empty you could safely
stop ?

Tom

> --
> Kohsuke Kawaguchi
> Sun Microsystems                   [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: spawning processes with Ant

mitch gottlieb-2
To add to that: I was just calling a batch file that was starting a java
process as a service (java wrapper) and Hudson still considered that as a
process still running.


-----Original Message-----
From: Tom Huybrechts [mailto:[hidden email]]
Sent: Wednesday, July 04, 2007 6:22 AM
To: [hidden email]
Subject: Re: spawning processes with Ant

On 7/4/07, Kohsuke Kawaguchi <[hidden email]> wrote:

> Tom Huybrechts wrote:
> > The first part of starting the new server, is shutting down any
> > running instances, so the number of processes is not a problem.
> >
> > If I understand correctly, the problem is that Hudson waits for the
> > stdout and stderr handles to close. Would it be possible to make this
> > configurable, and have Hudson stop the build when the process ends ?
>
> I guess it's possible. The problem is that Hudson cannot reliably
> capture all the output from the process if we do that.
>
> It's possible that the process writes a lot of stuff (which gets
> buffered in the pipe), then quit. If we consider the build over as soon
> as the process terminates, then we might not be able to drain the buffer.
>
>
> I think it's important to point out that the real problem is in those
> programs that get spawned. If they take proper measures to be a real
> daemon, this problem won't happen. Hudson blocks forever because
> stdout/stderr pipes between Hudson and the child process are still open,
> and that's because the daemon process inherits them and forgot to close
> them.
>
> It looks like there are tools that you can use to work around this
> issue: http://www.clapper.org/software/daemonize/
>
> We should at least put this on Wiki.
>

Unfortunately I'm on Windows, so that won't work for me. There are
tools that let your app run like a Windows service (srvany), but
that's a bit overkill. I thought nohup.exe would do the trick, but
that doesn't work either.

Wouldn't this be a workable solution ?
- provide an option to ignore child processes,
- wait for the process to finish,
- and then wait for the buffers to be drained,

If you ignore child processes, then nothing can be added to the
buffers after the process ends, so once it is empty you could safely
stop ?

Tom

> --
> Kohsuke Kawaguchi
> Sun Microsystems                   [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: spawning processes with Ant

Kohsuke Kawaguchi
Administrator
In reply to this post by Tom Huybrechts
Tom Huybrechts wrote:
> Wouldn't this be a workable solution ?
> - provide an option to ignore child processes,
> - wait for the process to finish,
> - and then wait for the buffers to be drained,

AFAIK, there's no way to properly "wait for the buffer to be drained."

It's InputStream, so we have to read and block w/o timeout nor
interruption. There's no NIO support for process either.

> If you ignore child processes, then nothing can be added to the
> buffers after the process ends, so once it is empty you could safely
> stop ?

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: spawning processes with Ant

Kohsuke Kawaguchi
Administrator
In reply to this post by mitch gottlieb-2
mitch gottlieb wrote:
> To add to that: I was just calling a batch file that was starting a java
> process as a service (java wrapper) and Hudson still considered that as a
> process still running.

Right. Again, that's because file handles are inherited all the way down.

If you use a tool like process explorer, you can actually see the
offending file handle.


>
> -----Original Message-----
> From: Tom Huybrechts [mailto:[hidden email]]
> Sent: Wednesday, July 04, 2007 6:22 AM
> To: [hidden email]
> Subject: Re: spawning processes with Ant
>
> On 7/4/07, Kohsuke Kawaguchi <[hidden email]> wrote:
>> Tom Huybrechts wrote:
>> > The first part of starting the new server, is shutting down any
>> > running instances, so the number of processes is not a problem.
>> >
>> > If I understand correctly, the problem is that Hudson waits for the
>> > stdout and stderr handles to close. Would it be possible to make this
>> > configurable, and have Hudson stop the build when the process ends ?
>>
>> I guess it's possible. The problem is that Hudson cannot reliably
>> capture all the output from the process if we do that.
>>
>> It's possible that the process writes a lot of stuff (which gets
>> buffered in the pipe), then quit. If we consider the build over as soon
>> as the process terminates, then we might not be able to drain the buffer.
>>
>>
>> I think it's important to point out that the real problem is in those
>> programs that get spawned. If they take proper measures to be a real
>> daemon, this problem won't happen. Hudson blocks forever because
>> stdout/stderr pipes between Hudson and the child process are still open,
>> and that's because the daemon process inherits them and forgot to close
>> them.
>>
>> It looks like there are tools that you can use to work around this
>> issue: http://www.clapper.org/software/daemonize/
>>
>> We should at least put this on Wiki.
>>
>
> Unfortunately I'm on Windows, so that won't work for me. There are
> tools that let your app run like a Windows service (srvany), but
> that's a bit overkill. I thought nohup.exe would do the trick, but
> that doesn't work either.
>
> Wouldn't this be a workable solution ?
> - provide an option to ignore child processes,
> - wait for the process to finish,
> - and then wait for the buffers to be drained,
>
> If you ignore child processes, then nothing can be added to the
> buffers after the process ends, so once it is empty you could safely
> stop ?
>
> Tom
>
>> --
>> Kohsuke Kawaguchi
>> Sun Microsystems                   [hidden email]
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: spawning processes with Ant

Tom Huybrechts
In reply to this post by mitch gottlieb-2
Just for the record: I 'fixed' this in my Windows/Ant build using:

        <scriptdef name="get-next-minute" language="beanshell">
                <attribute name="property" />
                date = new java.text.SimpleDateFormat("HH:mm").format(new
Date(System.currentTimeMillis() + 60000));
                project.setProperty(attributes.get("property"), date);
        </scriptdef>

                <get-next-minute property="next-minute" />
                <exec executable="at">
                        <arg value="${next-minute}" />
                        <arg value="/interactive" />
                        <arg value="${jboss.home}\bin\run.bat" />
                </exec>


On 7/4/07, mitch gottlieb <[hidden email]> wrote:

> To add to that: I was just calling a batch file that was starting a java
> process as a service (java wrapper) and Hudson still considered that as a
> process still running.
>
>
> -----Original Message-----
> From: Tom Huybrechts [mailto:[hidden email]]
> Sent: Wednesday, July 04, 2007 6:22 AM
> To: [hidden email]
> Subject: Re: spawning processes with Ant
>
> On 7/4/07, Kohsuke Kawaguchi <[hidden email]> wrote:
> > Tom Huybrechts wrote:
> > > The first part of starting the new server, is shutting down any
> > > running instances, so the number of processes is not a problem.
> > >
> > > If I understand correctly, the problem is that Hudson waits for the
> > > stdout and stderr handles to close. Would it be possible to make this
> > > configurable, and have Hudson stop the build when the process ends ?
> >
> > I guess it's possible. The problem is that Hudson cannot reliably
> > capture all the output from the process if we do that.
> >
> > It's possible that the process writes a lot of stuff (which gets
> > buffered in the pipe), then quit. If we consider the build over as soon
> > as the process terminates, then we might not be able to drain the buffer.
> >
> >
> > I think it's important to point out that the real problem is in those
> > programs that get spawned. If they take proper measures to be a real
> > daemon, this problem won't happen. Hudson blocks forever because
> > stdout/stderr pipes between Hudson and the child process are still open,
> > and that's because the daemon process inherits them and forgot to close
> > them.
> >
> > It looks like there are tools that you can use to work around this
> > issue: http://www.clapper.org/software/daemonize/
> >
> > We should at least put this on Wiki.
> >
>
> Unfortunately I'm on Windows, so that won't work for me. There are
> tools that let your app run like a Windows service (srvany), but
> that's a bit overkill. I thought nohup.exe would do the trick, but
> that doesn't work either.
>
> Wouldn't this be a workable solution ?
> - provide an option to ignore child processes,
> - wait for the process to finish,
> - and then wait for the buffers to be drained,
>
> If you ignore child processes, then nothing can be added to the
> buffers after the process ends, so once it is empty you could safely
> stop ?
>
> Tom
>
> > --
> > Kohsuke Kawaguchi
> > Sun Microsystems                   [hidden email]
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: spawning processes with Ant

Kohsuke Kawaguchi
Administrator

Thanks. I captured that in the Wiki.

I wonder if Windows "at" command supports "now" token (on Unix, it
does.) That would be convenient as you can avoid computing "the next
minute".

Tom Huybrechts wrote:

> Just for the record: I 'fixed' this in my Windows/Ant build using:
>
> <scriptdef name="get-next-minute" language="beanshell">
> <attribute name="property" />
> date = new java.text.SimpleDateFormat("HH:mm").format(new
> Date(System.currentTimeMillis() + 60000));
> project.setProperty(attributes.get("property"), date);
> </scriptdef>
>
> <get-next-minute property="next-minute" />
> <exec executable="at">
> <arg value="${next-minute}" />
> <arg value="/interactive" />
> <arg value="${jboss.home}\bin\run.bat" />
> </exec>

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: spawning processes with Ant

Tom Huybrechts
I don't think it supports 'now', and if you schedule it at the current
time, it's only executed a day later. However, I found an even better
solution using Windows Scripting Host (which is installed by default
on Windows 98+).

You need the attached script, and can run your program as follows.

<exec executable="cscript.exe">
   <env key="ANTRUN_TITLE" value="Title for Window" />  <!-- optional -->
   <env key="ANTRUN_OUTPUT" value="output.log" />  <!-- optional -->
   <arg value="//NoLogo" />
   <arg value="antRunAsync.js" />  <!-- this script -->
   <arg value="real executable" />
</exec>


On 7/7/07, Kohsuke Kawaguchi <[hidden email]> wrote:

>
> Thanks. I captured that in the Wiki.
>
> I wonder if Windows "at" command supports "now" token (on Unix, it
> does.) That would be convenient as you can avoid computing "the next
> minute".
>
> Tom Huybrechts wrote:
> > Just for the record: I 'fixed' this in my Windows/Ant build using:
> >
> >       <scriptdef name="get-next-minute" language="beanshell">
> >               <attribute name="property" />
> >               date = new java.text.SimpleDateFormat("HH:mm").format(new
> > Date(System.currentTimeMillis() + 60000));
> >               project.setProperty(attributes.get("property"), date);
> >       </scriptdef>
> >
> >               <get-next-minute property="next-minute" />
> >               <exec executable="at">
> >                       <arg value="${next-minute}" />
> >                       <arg value="/interactive" />
> >                       <arg value="${jboss.home}\bin\run.bat" />
> >               </exec>
>
>
> --
> Kohsuke Kawaguchi
> Sun Microsystems                   [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

antRunAsync.js (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: spawning processes with Ant

Kohsuke Kawaguchi
Administrator
Tom Huybrechts wrote:
> I don't think it supports 'now', and if you schedule it at the current
> time, it's only executed a day later. However, I found an even better
> solution using Windows Scripting Host (which is installed by default
> on Windows 98+).
>
> You need the attached script, and can run your program as follows.

Thanks. Also captured in Wiki.

--
Kohsuke Kawaguchi
Sun Microsystems                   [hidden email]

smime.p7s (4K) Download Attachment