revision.txt in one build directory, other files in another

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

revision.txt in one build directory, other files in another

Rhett Sutphin
Hi,

I've got two nearly identical ant builds in my hudson deployment.  
They point at the same subversion repository and poll at the same  
intervals.  A few days ago, one of them stopped picking up changes.  
It continues to poll (i.e., the subversion polling log changes every  
few minutes), but it never finds any changes and so never builds.  
The other build is still working fine, polling and building when  
there are changes.

Another symptom of this problem is that manual builds work, and the  
build execution correctly does the subversion update, but the build  
logs do not include the changes between these builds.

Poking around at the build logs, I found that the build after the  
last build that has changes recorded (and which I think is the last  
build which ran from polling) has its revision.txt in a separate  
directory from the rest of the build files.  Specifically:

builds$ ls 2007-11-29_19-31*
2007-11-29_19-31-24:
total 952
drwxr-xr-x     5 build  buildadm     170 Nov 29 19:32 .
drwxr-xr-x   595 build  buildadm   20230 Dec  5 14:13 ..
-rw-r--r--     1 build  buildadm     816 Nov 29 19:32 build.xml
-rw-r--r--     1 build  buildadm      44 Nov 29 19:31 changelog.xml
-rw-r--r--     1 build  buildadm  478665 Nov 29 19:32 log

2007-11-29_19-31-25:
total 8
drwxr-xr-x     3 build  buildadm    102 Nov 29 19:31 .
drwxr-xr-x   595 build  buildadm  20230 Dec  5 14:13 ..
-rw-r--r--     1 build  buildadm     69 Nov 29 19:31 revision.txt

Subsequent to these two, there are four empty build directories.  The  
timestamp on each one matches up approximately with a commit in the  
underlying repository, but there is not one for every change to the  
repository.  (Specifically, there were 12 builds of the parallel  
still-working job over the time span covered by these four empty  
directories).  Casually looking through my other build logs finds no  
other empty build directories, so I assume the empty directories is  
also a symptom.

After this, there are a mixture of correct (revision.txt in same  
directory as everything else), incorrect (revision.txt in separate  
directory), and empty build directories.  In each of the incorrect  
cases, the directory with the revision.txt is timestamped within 3  
seconds of the corresponding other directory.  Sometimes the  
revision.txt directory comes first and sometimes the other one comes  
first.  (These subsequent logs all are from manually kicked-off  
builds, of course.)

I've taken the hudson server down and back up to see if that would  
restore correct behavior (it didn't). There's nothing in the build  
logs that indicates to me what caused the original problem, but I'd  
be happy to supply them if someone else would care to take a look.  
This problem originated under 1.151.  Updating to 1.160 does not seem  
to have fixed it.

I'm not submitting this as a bug because I have no idea how to  
reproduce it.  I am mentioning it here because it seems like a subtle  
race condition (maybe) and therefore worth possibly investigating.  
Please let me know if there's anything other information I can provide.

Rhett

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: revision.txt in one build directory, other files in another

Kohsuke Kawaguchi
Administrator
2007/12/5, Rhett Sutphin <[hidden email]>:

> builds$ ls 2007-11-29_19-31*
> 2007-11-29_19-31-24:
> total 952
> drwxr-xr-x     5 build  buildadm     170 Nov 29 19:32 .
> drwxr-xr-x   595 build  buildadm   20230 Dec  5 14:13 ..
> -rw-r--r--     1 build  buildadm     816 Nov 29 19:32 build.xml
> -rw-r--r--     1 build  buildadm      44 Nov 29 19:31 changelog.xml
> -rw-r--r--     1 build  buildadm  478665 Nov 29 19:32 log
>
> 2007-11-29_19-31-25:
> total 8
> drwxr-xr-x     3 build  buildadm    102 Nov 29 19:31 .
> drwxr-xr-x   595 build  buildadm  20230 Dec  5 14:13 ..
> -rw-r--r--     1 build  buildadm     69 Nov 29 19:31 revision.txt

I notice that the timestamp of log and buil.xml is after the
revision.txt timestamp, so there's definitely something wrong.

Looking at the code, I'm not sure exactly how this happens. The only
feasible explanation is that somehow the Calendar instance that a
Build object holds to is modified, but I didn't find any such code.

I guess it's probably not a good idea for a build to retain a Calendar
object, as it is mutable. I guess I should switch to just 'long'.

Can you file an RFE for that? (or if you'd be interested in
implementing that by yourself that would be even better)


> Subsequent to these two, there are four empty build directories.  The
> timestamp on each one matches up approximately with a commit in the
> underlying repository, but there is not one for every change to the
> repository.  (Specifically, there were 12 builds of the parallel
> still-working job over the time span covered by these four empty
> directories).  Casually looking through my other build logs finds no
> other empty build directories, so I assume the empty directories is
> also a symptom.
>
> After this, there are a mixture of correct (revision.txt in same
> directory as everything else), incorrect (revision.txt in separate
> directory), and empty build directories.  In each of the incorrect
> cases, the directory with the revision.txt is timestamped within 3
> seconds of the corresponding other directory.  Sometimes the
> revision.txt directory comes first and sometimes the other one comes
> first.  (These subsequent logs all are from manually kicked-off
> builds, of course.)
>
> I've taken the hudson server down and back up to see if that would
> restore correct behavior (it didn't). There's nothing in the build
> logs that indicates to me what caused the original problem, but I'd
> be happy to supply them if someone else would care to take a look.
> This problem originated under 1.151.  Updating to 1.160 does not seem
> to have fixed it.
>
> I'm not submitting this as a bug because I have no idea how to
> reproduce it.  I am mentioning it here because it seems like a subtle
> race condition (maybe) and therefore worth possibly investigating.
> Please let me know if there's anything other information I can provide.

--
Kohsuke Kawaguchi

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: revision.txt in one build directory, other files in another

Rhett Sutphin
On Dec 6, 2007, at 1:13 AM, Kohsuke Kawaguchi wrote:

> 2007/12/5, Rhett Sutphin <[hidden email]>:
>> builds$ ls 2007-11-29_19-31*
>> 2007-11-29_19-31-24:
>> total 952
>> drwxr-xr-x     5 build  buildadm     170 Nov 29 19:32 .
>> drwxr-xr-x   595 build  buildadm   20230 Dec  5 14:13 ..
>> -rw-r--r--     1 build  buildadm     816 Nov 29 19:32 build.xml
>> -rw-r--r--     1 build  buildadm      44 Nov 29 19:31 changelog.xml
>> -rw-r--r--     1 build  buildadm  478665 Nov 29 19:32 log
>>
>> 2007-11-29_19-31-25:
>> total 8
>> drwxr-xr-x     3 build  buildadm    102 Nov 29 19:31 .
>> drwxr-xr-x   595 build  buildadm  20230 Dec  5 14:13 ..
>> -rw-r--r--     1 build  buildadm     69 Nov 29 19:31 revision.txt
>
> I notice that the timestamp of log and buil.xml is after the
> revision.txt timestamp, so there's definitely something wrong.
>
> Looking at the code, I'm not sure exactly how this happens. The only
> feasible explanation is that somehow the Calendar instance that a
> Build object holds to is modified, but I didn't find any such code.
>
> I guess it's probably not a good idea for a build to retain a Calendar
> object, as it is mutable. I guess I should switch to just 'long'.
>
> Can you file an RFE for that? (or if you'd be interested in
> implementing that by yourself that would be even better)

Done: https://hudson.dev.java.net/issues/show_bug.cgi?id=1084

I wouldn't mind taking a look at it myself, except that (last time I  
checked) hudson would only build on Java 6.  I develop on OS X, so  
that's a no go for me for the moment.

And as a final note:  I resolved the problem for this particular job  
by stopping hudson and purging all the build logs back to the first  
one that showed the problem.  When I restarted hudson, that job  
polled as usual and found all (103) commits that had happened in the  
meantime.

Rhett

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: revision.txt in one build directory, other files in another

Kohsuke Kawaguchi
Administrator
Thanks. I've heard that some had success running JDK6 on Mac OS X
using http://landonf.bikemonkey.org/static/soylatte

2007/12/10, Rhett Sutphin <[hidden email]>:

> On Dec 6, 2007, at 1:13 AM, Kohsuke Kawaguchi wrote:
> > 2007/12/5, Rhett Sutphin <[hidden email]>:
> >> builds$ ls 2007-11-29_19-31*
> >> 2007-11-29_19-31-24:
> >> total 952
> >> drwxr-xr-x     5 build  buildadm     170 Nov 29 19:32 .
> >> drwxr-xr-x   595 build  buildadm   20230 Dec  5 14:13 ..
> >> -rw-r--r--     1 build  buildadm     816 Nov 29 19:32 build.xml
> >> -rw-r--r--     1 build  buildadm      44 Nov 29 19:31 changelog.xml
> >> -rw-r--r--     1 build  buildadm  478665 Nov 29 19:32 log
> >>
> >> 2007-11-29_19-31-25:
> >> total 8
> >> drwxr-xr-x     3 build  buildadm    102 Nov 29 19:31 .
> >> drwxr-xr-x   595 build  buildadm  20230 Dec  5 14:13 ..
> >> -rw-r--r--     1 build  buildadm     69 Nov 29 19:31 revision.txt
> >
> > I notice that the timestamp of log and buil.xml is after the
> > revision.txt timestamp, so there's definitely something wrong.
> >
> > Looking at the code, I'm not sure exactly how this happens. The only
> > feasible explanation is that somehow the Calendar instance that a
> > Build object holds to is modified, but I didn't find any such code.
> >
> > I guess it's probably not a good idea for a build to retain a Calendar
> > object, as it is mutable. I guess I should switch to just 'long'.
> >
> > Can you file an RFE for that? (or if you'd be interested in
> > implementing that by yourself that would be even better)
>
> Done: https://hudson.dev.java.net/issues/show_bug.cgi?id=1084
>
> I wouldn't mind taking a look at it myself, except that (last time I
> checked) hudson would only build on Java 6.  I develop on OS X, so
> that's a no go for me for the moment.
>
> And as a final note:  I resolved the problem for this particular job
> by stopping hudson and purging all the build logs back to the first
> one that showed the problem.  When I restarted hudson, that job
> polled as usual and found all (103) commits that had happened in the
> meantime.
>
> Rhett
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
Kohsuke Kawaguchi

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: revision.txt in one build directory, other files in another

Joshua McKinnon
I'll second that and add that I've done this myself on my Macbook Pro.
I compiled Landon's JDK6 with no issues - It only took around an hour
to build. I was surprised how simple it was.

On 12/16/07, Kohsuke Kawaguchi <[hidden email]> wrote:

> Thanks. I've heard that some had success running JDK6 on Mac OS X
> using http://landonf.bikemonkey.org/static/soylatte
>
> 2007/12/10, Rhett Sutphin <[hidden email]>:
> > On Dec 6, 2007, at 1:13 AM, Kohsuke Kawaguchi wrote:
> > > 2007/12/5, Rhett Sutphin <[hidden email]>:
> > >> builds$ ls 2007-11-29_19-31*
> > >> 2007-11-29_19-31-24:
> > >> total 952
> > >> drwxr-xr-x     5 build  buildadm     170 Nov 29 19:32 .
> > >> drwxr-xr-x   595 build  buildadm   20230 Dec  5 14:13 ..
> > >> -rw-r--r--     1 build  buildadm     816 Nov 29 19:32 build.xml
> > >> -rw-r--r--     1 build  buildadm      44 Nov 29 19:31 changelog.xml
> > >> -rw-r--r--     1 build  buildadm  478665 Nov 29 19:32 log
> > >>
> > >> 2007-11-29_19-31-25:
> > >> total 8
> > >> drwxr-xr-x     3 build  buildadm    102 Nov 29 19:31 .
> > >> drwxr-xr-x   595 build  buildadm  20230 Dec  5 14:13 ..
> > >> -rw-r--r--     1 build  buildadm     69 Nov 29 19:31 revision.txt
> > >
> > > I notice that the timestamp of log and buil.xml is after the
> > > revision.txt timestamp, so there's definitely something wrong.
> > >
> > > Looking at the code, I'm not sure exactly how this happens. The only
> > > feasible explanation is that somehow the Calendar instance that a
> > > Build object holds to is modified, but I didn't find any such code.
> > >
> > > I guess it's probably not a good idea for a build to retain a Calendar
> > > object, as it is mutable. I guess I should switch to just 'long'.
> > >
> > > Can you file an RFE for that? (or if you'd be interested in
> > > implementing that by yourself that would be even better)
> >
> > Done: https://hudson.dev.java.net/issues/show_bug.cgi?id=1084
> >
> > I wouldn't mind taking a look at it myself, except that (last time I
> > checked) hudson would only build on Java 6.  I develop on OS X, so
> > that's a no go for me for the moment.
> >
> > And as a final note:  I resolved the problem for this particular job
> > by stopping hudson and purging all the build logs back to the first
> > one that showed the problem.  When I restarted hudson, that job
> > polled as usual and found all (103) commits that had happened in the
> > meantime.
> >
> > Rhett
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
>
> --
> Kohsuke Kawaguchi
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Loading...