[Issue 1989] New - Builds start 12 1/2 hours late

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[Issue 1989] New - Builds start 12 1/2 hours late

johmart
https://hudson.dev.java.net/issues/show_bug.cgi?id=1989
                 Issue #|1989
                 Summary|Builds start 12 1/2 hours late
               Component|hudson
                 Version|current
                Platform|Other
              OS/Version|Linux
                     URL|
                  Status|NEW
       Status whiteboard|
                Keywords|
              Resolution|
              Issue type|DEFECT
                Priority|P3
            Subcomponent|other
             Assigned to|issues@hudson
             Reported by|johmart






------- Additional comments from [hidden email] Mon Jun 30 07:44:28 +0000 2008 -------
I had two builds scheduled for Saturday morning using cron expressions:
0 9 * * 6
and
0 10 * * 6

Hudson was idle all day and started both jobs on Saturday at 22:31:26.

The version of hudson deployed is 1227. Hudson is running inside JBoss 4.2.2,
jdk1.5.0_15 (64bit) inside a OpenSUSE 10.3 virtual machine (VMWare host).

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 1989] Builds start 12 1/2 hours late

Kohsuke Kawaguchi
Administrator
https://hudson.dev.java.net/issues/show_bug.cgi?id=1989



User kohsuke changed the following:

                What    |Old value                 |New value
================================================================================
                  Status|NEW                       |STARTED
--------------------------------------------------------------------------------




------- Additional comments from [hidden email] Wed Jul  2 02:04:03 +0000 2008 -------
In 1.232 I tweaked the logging code so that you can see this behavior more in
the logger.

Can you enable the CONFIG level logging for "hudson.triggers.Trigger" logger and
see when the builds are triggered?

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 1989] Builds start 12 1/2 hours late

scm_issue_link
In reply to this post by johmart
https://hudson.dev.java.net/issues/show_bug.cgi?id=1989






------- Additional comments from [hidden email] Wed Jul  2 03:24:04 +0000 2008 -------
Code changed in hudson
User: : kohsuke
Path:
 trunk/hudson/main/core/src/main/java/hudson/triggers/Trigger.java
http://fisheye4.cenqua.com/changelog/hudson/?cs=10570
Log:
improving the logging level to help assist HUDSON-1989.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 1989] Builds start 12 1/2 hours late

scm_issue_link
In reply to this post by johmart
https://hudson.dev.java.net/issues/show_bug.cgi?id=1989






------- Additional comments from [hidden email] Wed Jul  2 03:49:02 +0000 2008 -------
Code changed in hudson
User: : kohsuke
Path:
 trunk/hudson/main/core/src/main/java/hudson/triggers/Trigger.java
http://fisheye4.cenqua.com/changelog/hudson/?cs=10570
Log:
improving the logging level to help assist HUDSON-1989.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 1989] Builds start 12 1/2 hours late

scm_issue_link
In reply to this post by johmart
https://hudson.dev.java.net/issues/show_bug.cgi?id=1989






------- Additional comments from [hidden email] Wed Jul  2 04:17:26 +0000 2008 -------
Code changed in hudson
User: : kohsuke
Path:
 trunk/hudson/main/core/src/main/java/hudson/triggers/Trigger.java
http://fisheye4.cenqua.com/changelog/hudson/?cs=10570
Log:
improving the logging level to help assist HUDSON-1989.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 1989] Builds start 12 1/2 hours late

scm_issue_link
In reply to this post by johmart
https://hudson.dev.java.net/issues/show_bug.cgi?id=1989






------- Additional comments from [hidden email] Wed Jul  2 04:49:02 +0000 2008 -------
Code changed in hudson
User: : kohsuke
Path:
 trunk/hudson/main/core/src/main/java/hudson/triggers/Trigger.java
http://fisheye4.cenqua.com/changelog/hudson/?cs=10570
Log:
improving the logging level to help assist HUDSON-1989.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 1989] Builds start 12 1/2 hours late

scm_issue_link
In reply to this post by johmart
https://hudson.dev.java.net/issues/show_bug.cgi?id=1989






------- Additional comments from [hidden email] Wed Jul  2 05:24:08 +0000 2008 -------
Code changed in hudson
User: : kohsuke
Path:
 trunk/hudson/main/core/src/main/java/hudson/triggers/Trigger.java
http://fisheye4.cenqua.com/changelog/hudson/?cs=10570
Log:
improving the logging level to help assist HUDSON-1989.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 1989] Builds start 12 1/2 hours late

johmart
In reply to this post by johmart
https://hudson.dev.java.net/issues/show_bug.cgi?id=1989






------- Additional comments from [hidden email] Wed Jul  2 05:59:15 +0000 2008 -------
I noticed something else that might be connected: for another job, I had setup
subversion polling for "55 * * * *". According to the polling log, polling stops
every night at midnight.

The virtual machine runs a script to adjust the system time every night at
midnight. The virtual hardware clock was off by about 1 1/2 hours, so the time
set wasn't right. VMWare tools corrected the time again about a minute later. So
time around midnight was very non-linear, kind of like this:
* midnight: time is 00:00:00
* cron starts clock script
* time becomes 22:31:56 the day before
* about 1-2 minutes pass
* vmware-tools sets time to 00:02:00 (right day)

I've fixed the virtual hardware clock for now, but the problem will reoccur in a
milder variant since the system clock inside a 64-bit virtual machine in vmware
always runs fast, it gains a few seconds every day. So around midnight, the time
will become non-linear.

vmware-tools is supposed to take care of this, but it only fixes clocks that are
slow (such as in a 32-bit virtual machine), not those that are fast. That's why
I need to run the script to synchronize the time with the hardware clock.

I've upgraded to 1.231 (1.232 is not yet available for download) and set the
loglevel for hudson.triggers.Trigger to "all".

I just recreated the scenario that occurs around midnight, I don't see any more
trigger entries in the hudson log after the point where the time was first reset.





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 1989] Builds start 12 1/2 hours late

scm_issue_link
In reply to this post by johmart
https://hudson.dev.java.net/issues/show_bug.cgi?id=1989






------- Additional comments from [hidden email] Wed Jul  2 06:11:26 +0000 2008 -------
Code changed in hudson
User: : kohsuke
Path:
 trunk/hudson/main/core/src/main/java/hudson/triggers/Trigger.java
http://fisheye4.cenqua.com/changelog/hudson/?cs=10570
Log:
improving the logging level to help assist HUDSON-1989.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 1989] Builds start 12 1/2 hours late

scm_issue_link
In reply to this post by johmart
https://hudson.dev.java.net/issues/show_bug.cgi?id=1989






------- Additional comments from [hidden email] Wed Jul  2 06:44:05 +0000 2008 -------
Code changed in hudson
User: : kohsuke
Path:
 trunk/hudson/main/core/src/main/java/hudson/triggers/Trigger.java
http://fisheye4.cenqua.com/changelog/hudson/?cs=10570
Log:
improving the logging level to help assist HUDSON-1989.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 1989] Builds start 12 1/2 hours late

scm_issue_link
In reply to this post by johmart
https://hudson.dev.java.net/issues/show_bug.cgi?id=1989






------- Additional comments from [hidden email] Wed Jul  2 07:29:11 +0000 2008 -------
Code changed in hudson
User: : kohsuke
Path:
 trunk/hudson/main/core/src/main/java/hudson/triggers/Trigger.java
http://fisheye4.cenqua.com/changelog/hudson/?cs=10570
Log:
improving the logging level to help assist HUDSON-1989.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 1989] Builds start 12 1/2 hours late

scm_issue_link
In reply to this post by johmart
https://hudson.dev.java.net/issues/show_bug.cgi?id=1989






------- Additional comments from [hidden email] Wed Jul  2 08:17:53 +0000 2008 -------
Code changed in hudson
User: : kohsuke
Path:
 trunk/hudson/main/core/src/main/java/hudson/triggers/Trigger.java
http://fisheye4.cenqua.com/changelog/hudson/?cs=10570
Log:
improving the logging level to help assist HUDSON-1989.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 1989] Builds start 12 1/2 hours late

scm_issue_link
In reply to this post by johmart
https://hudson.dev.java.net/issues/show_bug.cgi?id=1989






------- Additional comments from [hidden email] Wed Jul  2 09:09:30 +0000 2008 -------
Code changed in hudson
User: : kohsuke
Path:
 trunk/hudson/main/core/src/main/java/hudson/triggers/Trigger.java
http://fisheye4.cenqua.com/changelog/hudson/?cs=10570
Log:
improving the logging level to help assist HUDSON-1989.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 1989] Builds start 12 1/2 hours late

scm_issue_link
In reply to this post by johmart
https://hudson.dev.java.net/issues/show_bug.cgi?id=1989






------- Additional comments from [hidden email] Wed Jul  2 10:05:06 +0000 2008 -------
Code changed in hudson
User: : kohsuke
Path:
 trunk/hudson/main/core/src/main/java/hudson/triggers/Trigger.java
http://fisheye4.cenqua.com/changelog/hudson/?cs=10570
Log:
improving the logging level to help assist HUDSON-1989.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 1989] Builds start 12 1/2 hours late

scm_issue_link
In reply to this post by johmart
https://hudson.dev.java.net/issues/show_bug.cgi?id=1989






------- Additional comments from [hidden email] Wed Jul  2 11:02:49 +0000 2008 -------
Code changed in hudson
User: : kohsuke
Path:
 trunk/hudson/main/core/src/main/java/hudson/triggers/Trigger.java
http://fisheye4.cenqua.com/changelog/hudson/?cs=10570
Log:
improving the logging level to help assist HUDSON-1989.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 1989] Builds start 12 1/2 hours late

scm_issue_link
In reply to this post by johmart
https://hudson.dev.java.net/issues/show_bug.cgi?id=1989






------- Additional comments from [hidden email] Wed Jul  2 12:04:47 +0000 2008 -------
Code changed in hudson
User: : kohsuke
Path:
 trunk/hudson/main/core/src/main/java/hudson/triggers/Trigger.java
http://fisheye4.cenqua.com/changelog/hudson/?cs=10570
Log:
improving the logging level to help assist HUDSON-1989.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 1989] Builds start 12 1/2 hours late

scm_issue_link
In reply to this post by johmart
https://hudson.dev.java.net/issues/show_bug.cgi?id=1989






------- Additional comments from [hidden email] Wed Jul  2 13:09:17 +0000 2008 -------
Code changed in hudson
User: : kohsuke
Path:
 trunk/hudson/main/core/src/main/java/hudson/triggers/Trigger.java
http://fisheye4.cenqua.com/changelog/hudson/?cs=10570
Log:
improving the logging level to help assist HUDSON-1989.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 1989] Builds start 12 1/2 hours late

scm_issue_link
In reply to this post by johmart
https://hudson.dev.java.net/issues/show_bug.cgi?id=1989






------- Additional comments from [hidden email] Wed Jul  2 14:17:40 +0000 2008 -------
Code changed in hudson
User: : kohsuke
Path:
 trunk/hudson/main/core/src/main/java/hudson/triggers/Trigger.java
http://fisheye4.cenqua.com/changelog/hudson/?cs=10570
Log:
improving the logging level to help assist HUDSON-1989.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 1989] Builds start 12 1/2 hours late

scm_issue_link
In reply to this post by johmart
https://hudson.dev.java.net/issues/show_bug.cgi?id=1989






------- Additional comments from [hidden email] Wed Jul  2 15:29:07 +0000 2008 -------
Code changed in hudson
User: : kohsuke
Path:
 trunk/hudson/main/core/src/main/java/hudson/triggers/Trigger.java
http://fisheye4.cenqua.com/changelog/hudson/?cs=10570
Log:
improving the logging level to help assist HUDSON-1989.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Issue 1989] Builds start 12 1/2 hours late

scm_issue_link
In reply to this post by johmart
https://hudson.dev.java.net/issues/show_bug.cgi?id=1989






------- Additional comments from [hidden email] Wed Jul  2 16:44:10 +0000 2008 -------
Code changed in hudson
User: : kohsuke
Path:
 trunk/hudson/main/core/src/main/java/hudson/triggers/Trigger.java
http://fisheye4.cenqua.com/changelog/hudson/?cs=10570
Log:
improving the logging level to help assist HUDSON-1989.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

12