[JIRA] Created: (HUDSON-8408) Large number of jobs triggered on Hudson restart

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[JIRA] Created: (HUDSON-8408) Large number of jobs triggered on Hudson restart

Kohsuke Kawaguchi
Administrator
Large number of jobs triggered on Hudson restart
------------------------------------------------

                 Key: HUDSON-8408
                 URL: http://issues.hudson-ci.org/browse/HUDSON-8408
             Project: Hudson
          Issue Type: Bug
          Components: tfs
    Affects Versions: current
         Environment: Windows 2008 R2 build farm.  1 master, 7 slaves.  
            Reporter: tdiz
            Assignee: redsolo


Hi Tom

I havent heard of any similar issue so it would be great if you could
add it to the jira so it can be tracked.

Are all builds triggered by a SCM change? You can tell by looking at
one of the builds page, and if it says "triggered by SCM".
Are the jobs built on the same machine?

The build is started either because the workspace is invalid (or not
present on the machine) or if there has been a commit during a period.
Newer Hudson versions store the SCM polling log together with the
build (if it was started by a SCm change), so hopefully we can get
some info from that. To see the polling log for a certain build, go to
the Build page, click on the "Started by a SCM change and you should
see the full log. (similar to
http://ramfelt.se/job/Mockito/528/pollingLog/?)

If you dont have a link for the SCM change then you will have to
manually watch the SCM polling log just after you reboot your server
to see why the plugin triggers a new build.


Regards
//Erik
=========================================================================================
On Tue, Jan 4, 2011 at 13:57, Tom wrote:
> Good morning,
>
> We have a large Hudson master/slave farm using both the TFS and base clear
> case SCM plugins.   We've been trying to figure out why each time we restart
> Hudson (usually to install a plugin), Hudson triggers a lot of builds.   Not
> all, but a large number.    This morning I restarted, and looking at the
> list, realized all of the builds it's triggering on restart are TFS
> based.      Some blow away the workspace, some do not.  

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.hudson-ci.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[JIRA] Commented: (HUDSON-8408) Large number of jobs triggered on Hudson restart

Kohsuke Kawaguchi
Administrator

    [ http://issues.hudson-ci.org/browse/HUDSON-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=144618#action_144618 ]

tdiz commented on HUDSON-8408:
------------------------------

Yes all of these builds show that they were started by an SCM change, and right above that, "no changes."

Looking through the links on the SCM change text though, I see this:

Polling Log
View as plain text

This page captures the polling log that triggered this build.

Started on Jan 4, 2011 6:59:44 AM
Workspace is offline.
Scheduling a new build to get a workspace.
Done. Took 15 ms
Changes found

What does workspace is offline mean?

Thanks again!

> Large number of jobs triggered on Hudson restart
> ------------------------------------------------
>
>                 Key: HUDSON-8408
>                 URL: http://issues.hudson-ci.org/browse/HUDSON-8408
>             Project: Hudson
>          Issue Type: Bug
>          Components: tfs
>    Affects Versions: current
>         Environment: Windows 2008 R2 build farm.  1 master, 7 slaves.  
>            Reporter: tdiz
>            Assignee: redsolo
>
> Hi Tom
> I havent heard of any similar issue so it would be great if you could
> add it to the jira so it can be tracked.
> Are all builds triggered by a SCM change? You can tell by looking at
> one of the builds page, and if it says "triggered by SCM".
> Are the jobs built on the same machine?
> The build is started either because the workspace is invalid (or not
> present on the machine) or if there has been a commit during a period.
> Newer Hudson versions store the SCM polling log together with the
> build (if it was started by a SCm change), so hopefully we can get
> some info from that. To see the polling log for a certain build, go to
> the Build page, click on the "Started by a SCM change and you should
> see the full log. (similar to
> http://ramfelt.se/job/Mockito/528/pollingLog/?)
> If you dont have a link for the SCM change then you will have to
> manually watch the SCM polling log just after you reboot your server
> to see why the plugin triggers a new build.
> Regards
> //Erik
> =========================================================================================
> On Tue, Jan 4, 2011 at 13:57, Tom wrote:
> > Good morning,
> >
> > We have a large Hudson master/slave farm using both the TFS and base clear
> > case SCM plugins.   We've been trying to figure out why each time we restart
> > Hudson (usually to install a plugin), Hudson triggers a lot of builds.   Not
> > all, but a large number.    This morning I restarted, and looking at the
> > list, realized all of the builds it's triggering on restart are TFS
> > based.      Some blow away the workspace, some do not.  

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.hudson-ci.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[JIRA] Commented: (HUDSON-8408) Large number of jobs triggered on Hudson restart

Kohsuke Kawaguchi
Administrator
In reply to this post by Kohsuke Kawaguchi

    [ http://issues.hudson-ci.org/browse/HUDSON-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=144626#action_144626 ]

redsolo commented on HUDSON-8408:
---------------------------------

Maybe HUDSON-1348 is connected to this issue. As we dont see any output from the TFS command line, we can assume that it isnt the TFS command line tool triggering the change.

Are you building the jobs on different slaves? Are they available when the server is restarted?

> Large number of jobs triggered on Hudson restart
> ------------------------------------------------
>
>                 Key: HUDSON-8408
>                 URL: http://issues.hudson-ci.org/browse/HUDSON-8408
>             Project: Hudson
>          Issue Type: Bug
>          Components: tfs
>    Affects Versions: current
>         Environment: Windows 2008 R2 build farm.  1 master, 7 slaves.  
>            Reporter: tdiz
>            Assignee: redsolo
>
> Hi Tom
> I havent heard of any similar issue so it would be great if you could
> add it to the jira so it can be tracked.
> Are all builds triggered by a SCM change? You can tell by looking at
> one of the builds page, and if it says "triggered by SCM".
> Are the jobs built on the same machine?
> The build is started either because the workspace is invalid (or not
> present on the machine) or if there has been a commit during a period.
> Newer Hudson versions store the SCM polling log together with the
> build (if it was started by a SCm change), so hopefully we can get
> some info from that. To see the polling log for a certain build, go to
> the Build page, click on the "Started by a SCM change and you should
> see the full log. (similar to
> http://ramfelt.se/job/Mockito/528/pollingLog/?)
> If you dont have a link for the SCM change then you will have to
> manually watch the SCM polling log just after you reboot your server
> to see why the plugin triggers a new build.
> Regards
> //Erik
> =========================================================================================
> On Tue, Jan 4, 2011 at 13:57, Tom wrote:
> > Good morning,
> >
> > We have a large Hudson master/slave farm using both the TFS and base clear
> > case SCM plugins.   We've been trying to figure out why each time we restart
> > Hudson (usually to install a plugin), Hudson triggers a lot of builds.   Not
> > all, but a large number.    This morning I restarted, and looking at the
> > list, realized all of the builds it's triggering on restart are TFS
> > based.      Some blow away the workspace, some do not.  

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.hudson-ci.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[JIRA] Commented: (HUDSON-8408) Large number of jobs triggered on Hudson restart

Kohsuke Kawaguchi
Administrator
In reply to this post by Kohsuke Kawaguchi

    [ http://issues.hudson-ci.org/browse/HUDSON-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=144627#action_144627 ]

tdiz commented on HUDSON-8408:
------------------------------

Maybe, that's the one I kept coming up with in my searches on this problem.   Not sure how to know for sure.

Here are the first two lines of the console log:

07:40:02  Started by an SCM change
07:40:02  Building remotely on xxx-xxx-3

So yes it's building on different slaves (using labels).    The slave machines and Hudson slave process are up when we're restarting Hudson on the master.   Wondering if there's some timing issue either:

1. Restarting the master, and while it's still establishing that the slaves are up and running, it kicks off jobs.  Or

2. Something happening as a result of the ~60 TFS jobs all attempting to fire up tf.exe to look for changes at the same time (either delay on the server, or delay on the TFS server).

Those are just guesses though.  

> Large number of jobs triggered on Hudson restart
> ------------------------------------------------
>
>                 Key: HUDSON-8408
>                 URL: http://issues.hudson-ci.org/browse/HUDSON-8408
>             Project: Hudson
>          Issue Type: Bug
>          Components: tfs
>    Affects Versions: current
>         Environment: Windows 2008 R2 build farm.  1 master, 7 slaves.  
>            Reporter: tdiz
>            Assignee: redsolo
>
> Hi Tom
> I havent heard of any similar issue so it would be great if you could
> add it to the jira so it can be tracked.
> Are all builds triggered by a SCM change? You can tell by looking at
> one of the builds page, and if it says "triggered by SCM".
> Are the jobs built on the same machine?
> The build is started either because the workspace is invalid (or not
> present on the machine) or if there has been a commit during a period.
> Newer Hudson versions store the SCM polling log together with the
> build (if it was started by a SCm change), so hopefully we can get
> some info from that. To see the polling log for a certain build, go to
> the Build page, click on the "Started by a SCM change and you should
> see the full log. (similar to
> http://ramfelt.se/job/Mockito/528/pollingLog/?)
> If you dont have a link for the SCM change then you will have to
> manually watch the SCM polling log just after you reboot your server
> to see why the plugin triggers a new build.
> Regards
> //Erik
> =========================================================================================
> On Tue, Jan 4, 2011 at 13:57, Tom wrote:
> > Good morning,
> >
> > We have a large Hudson master/slave farm using both the TFS and base clear
> > case SCM plugins.   We've been trying to figure out why each time we restart
> > Hudson (usually to install a plugin), Hudson triggers a lot of builds.   Not
> > all, but a large number.    This morning I restarted, and looking at the
> > list, realized all of the builds it's triggering on restart are TFS
> > based.      Some blow away the workspace, some do not.  

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.hudson-ci.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[JIRA] Commented: (HUDSON-8408) Large number of jobs triggered on Hudson restart

Kohsuke Kawaguchi
Administrator
In reply to this post by Kohsuke Kawaguchi

    [ http://issues.hudson-ci.org/browse/HUDSON-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=144629#action_144629 ]

redsolo commented on HUDSON-8408:
---------------------------------

Hudson should try to build the job on the latest node that was used for building, but if it cant find it or use any other it will need to create a new workspace to be able to determine if there is any cahnge. I am not sure how the polling works or not, if it requires to be used on the last node or not.

Is Hudson building the jobs on different slaves? ie, job A was last built on node X; after the restart will job A be built on node X or will is use node Y?


I know there has been some changes in the SCM API, I will look into them and see if they apply to this kind of issue.

What Hudson version are you using?

> Large number of jobs triggered on Hudson restart
> ------------------------------------------------
>
>                 Key: HUDSON-8408
>                 URL: http://issues.hudson-ci.org/browse/HUDSON-8408
>             Project: Hudson
>          Issue Type: Bug
>          Components: tfs
>    Affects Versions: current
>         Environment: Windows 2008 R2 build farm.  1 master, 7 slaves.  
>            Reporter: tdiz
>            Assignee: redsolo
>
> Hi Tom
> I havent heard of any similar issue so it would be great if you could
> add it to the jira so it can be tracked.
> Are all builds triggered by a SCM change? You can tell by looking at
> one of the builds page, and if it says "triggered by SCM".
> Are the jobs built on the same machine?
> The build is started either because the workspace is invalid (or not
> present on the machine) or if there has been a commit during a period.
> Newer Hudson versions store the SCM polling log together with the
> build (if it was started by a SCm change), so hopefully we can get
> some info from that. To see the polling log for a certain build, go to
> the Build page, click on the "Started by a SCM change and you should
> see the full log. (similar to
> http://ramfelt.se/job/Mockito/528/pollingLog/?)
> If you dont have a link for the SCM change then you will have to
> manually watch the SCM polling log just after you reboot your server
> to see why the plugin triggers a new build.
> Regards
> //Erik
> =========================================================================================
> On Tue, Jan 4, 2011 at 13:57, Tom wrote:
> > Good morning,
> >
> > We have a large Hudson master/slave farm using both the TFS and base clear
> > case SCM plugins.   We've been trying to figure out why each time we restart
> > Hudson (usually to install a plugin), Hudson triggers a lot of builds.   Not
> > all, but a large number.    This morning I restarted, and looking at the
> > list, realized all of the builds it's triggering on restart are TFS
> > based.      Some blow away the workspace, some do not.  

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.hudson-ci.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[JIRA] Commented: (HUDSON-8408) Large number of jobs triggered on Hudson restart

Kohsuke Kawaguchi
Administrator
In reply to this post by Kohsuke Kawaguchi

    [ http://issues.hudson-ci.org/browse/HUDSON-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=144630#action_144630 ]

tdiz commented on HUDSON-8408:
------------------------------

After restart is it sticking to the last slave - good question.  I'll have to set up a test and check for that.

Maybe I need to spend more time w/ core Hudson logging to look at what actually happens on a master restart, assuming there's a way to turn on more verbose logging.  

We're on 1.389 with the 1.11 TFS plugin.  

I don't think this is anything new, we've been seeing it since we got slaves hooked up about 6 months ago.  Wasn't a big problem when we had 10 build jobs set up in the system, but we're over 100 already and rapidly growing.

> Large number of jobs triggered on Hudson restart
> ------------------------------------------------
>
>                 Key: HUDSON-8408
>                 URL: http://issues.hudson-ci.org/browse/HUDSON-8408
>             Project: Hudson
>          Issue Type: Bug
>          Components: tfs
>    Affects Versions: current
>         Environment: Windows 2008 R2 build farm.  1 master, 7 slaves.  
>            Reporter: tdiz
>            Assignee: redsolo
>
> Hi Tom
> I havent heard of any similar issue so it would be great if you could
> add it to the jira so it can be tracked.
> Are all builds triggered by a SCM change? You can tell by looking at
> one of the builds page, and if it says "triggered by SCM".
> Are the jobs built on the same machine?
> The build is started either because the workspace is invalid (or not
> present on the machine) or if there has been a commit during a period.
> Newer Hudson versions store the SCM polling log together with the
> build (if it was started by a SCm change), so hopefully we can get
> some info from that. To see the polling log for a certain build, go to
> the Build page, click on the "Started by a SCM change and you should
> see the full log. (similar to
> http://ramfelt.se/job/Mockito/528/pollingLog/?)
> If you dont have a link for the SCM change then you will have to
> manually watch the SCM polling log just after you reboot your server
> to see why the plugin triggers a new build.
> Regards
> //Erik
> =========================================================================================
> On Tue, Jan 4, 2011 at 13:57, Tom wrote:
> > Good morning,
> >
> > We have a large Hudson master/slave farm using both the TFS and base clear
> > case SCM plugins.   We've been trying to figure out why each time we restart
> > Hudson (usually to install a plugin), Hudson triggers a lot of builds.   Not
> > all, but a large number.    This morning I restarted, and looking at the
> > list, realized all of the builds it's triggering on restart are TFS
> > based.      Some blow away the workspace, some do not.  

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.hudson-ci.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira