ConsoleLogFilter not working for builds on slaves

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

ConsoleLogFilter not working for builds on slaves

Tim Webster
Hi,

I've written a plugin that forwards console logs to a remote TCP log collector, using a ConsoleLogFilter.  It works perfectly for jobs running on master, but gets ignored for jobs running on slaves.

It's borrowing heavily from the Splunk plugin in that it 'tees' the console logs both to the existing logger output stream and a remote socket output stream.

Can anyone point out what I'm missing?  Plugin source is here: https://github.com/timwebster9/jenkins-tcp-log-fowarder-plugin

thanks :-)

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/a7de05bf-1aec-4902-99eb-a60bbf10a45d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: ConsoleLogFilter not working for builds on slaves

Jesse Glick-4
On Fri, Jan 19, 2018 at 7:54 AM, Tim Webster <[hidden email]> wrote:
> I've written a plugin that forwards console logs to a remote TCP log
> collector, using a ConsoleLogFilter.

I am not sure of your specific problem, but BTW JENKINS-38381 and
related issues propose a more solid foundation foundation for this
class of plugin.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3x%2Bnxss3wWApMHhvRxu2Dy1WRijY_AJ2fDSyvnYBAs-w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: ConsoleLogFilter not working for builds on slaves

Tim Webster
Basically I just wanted a simple plugin that could forward console logs to a remote collector.  The existing ones around are pretty specific the the collector/indexer (e.g. Splunk, ELK, Logstash, etc).

We're looking at using OK Log, which only requires a host and port to connect to: https://github.com/oklog/oklog

There are other ways to get the console log there, but they are clunky (e.g. downloading it after the job and forwarding it, etc).

The specific problem that I'm seeing is that if the build runs on a remote slave (I'm using Jenkins Swarm plugin agents), my ConsoleLogFilter.decorateLogger() extension/implementation isn't even being called.  If it runs on master it works fine.  Was wondering if I need to use a different extension point or something (e.g. BuildWrapper.decorateLogger() )?



On Friday, 19 January 2018 14:34:29 UTC, Jesse Glick wrote:
On Fri, Jan 19, 2018 at 7:54 AM, Tim Webster <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="K2B8JoogBAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">tim.w...@...> wrote:
> I've written a plugin that forwards console logs to a remote TCP log
> collector, using a ConsoleLogFilter.

I am not sure of your specific problem, but BTW JENKINS-38381 and
related issues propose a more solid foundation foundation for this
class of plugin.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/e6b27816-868b-4086-ba65-e84e5561af00%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: ConsoleLogFilter not working for builds on slaves

Jesse Glick-4
On Fri, Jan 19, 2018 at 9:47 AM, Tim Webster <[hidden email]> wrote:
> Was wondering if I need to use
> a different extension point or something

Not that I am aware of. Sounds like a bug offhand.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3hHv59nyHZrFsFav1%2BK%2B4xC28A%3D23_X-d3ug%3DKqrxSqw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: ConsoleLogFilter not working for builds on slaves

Tim Webster
Just realised I think it's something to do with it being a Pipeline job - a freestyle job seems to work on the remote slave.

So:
- Freestyle job works everywhere
- Pipeline job only on master.

Any ideas?

On Friday, 19 January 2018 15:05:08 UTC, Jesse Glick wrote:
On Fri, Jan 19, 2018 at 9:47 AM, Tim Webster <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="HcxHVTYiBAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">tim.w...@...> wrote:
> Was wondering if I need to use
> a different extension point or something

Not that I am aware of. Sounds like a bug offhand.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/5f51f993-9a38-42bd-90c0-2a9e309c8d9f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: ConsoleLogFilter not working for builds on slaves

Tim Webster
apologies - pipeline doesn't work on master either.  I will look at the pipeline support for plugins stuff...

Thanks

On Friday, 19 January 2018 16:34:19 UTC, Tim Webster wrote:
Just realised I think it's something to do with it being a Pipeline job - a freestyle job seems to work on the remote slave.

So:
- Freestyle job works everywhere
- Pipeline job only on master.

Any ideas?

On Friday, 19 January 2018 15:05:08 UTC, Jesse Glick wrote:
On Fri, Jan 19, 2018 at 9:47 AM, Tim Webster <[hidden email]> wrote:
> Was wondering if I need to use
> a different extension point or something

Not that I am aware of. Sounds like a bug offhand.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/0a0a53b9-c2fd-4f62-9869-8429377035c7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: ConsoleLogFilter not working for builds on slaves

Jesse Glick-4
On Fri, Jan 19, 2018 at 11:50 AM, Tim Webster <[hidden email]> wrote:
> pipeline doesn't work on master either.

No, `ConsoleLogFilter` is not supported for Pipeline currently. See
JENKINS-38381, some variant of which would be the preferred API for
adding this kind of functionality.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3b%2BbuMPHUvwexkVdUPjNH5B1zdbnRJ%3Dryukpxmk0c8Yg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: ConsoleLogFilter not working for builds on slaves

Markus Winter
Why is ConsoleLogFilter not supported in pipeline.
Timestamper plugin is also using a ConsoleLogFilter to add the
timestamps in pipeline
(https://github.com/jenkinsci/timestamper-plugin/blob/master/src/main/java/hudson/plugins/timestamper/pipeline/TimestamperStep.java)
Sure it has the limitation that it will not catch all the output of the
pipeline (some output from the pipeline workflow execution engine is
missed).
BTW, I've opened a PR in the logstash plugin that does a major
refactoring that adds an extension point for indexers. So adding
something to write to oklog would be quite easy.

And I also have implemented (not submitted) a true pipeline BuildWrapper
for it so you can do
logstash {
   node('label')
{
   sh ....
}
}

On 19.01.2018 18:24, Jesse Glick wrote:
> On Fri, Jan 19, 2018 at 11:50 AM, Tim Webster <[hidden email]> wrote:
>> pipeline doesn't work on master either.
> No, `ConsoleLogFilter` is not supported for Pipeline currently. See
> JENKINS-38381, some variant of which would be the preferred API for
> adding this kind of functionality.
>

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/485d5605-d085-de4a-3e2e-2640cfc6c363%40gmx.de.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: ConsoleLogFilter not working for builds on slaves

Jesse Glick-4
On Fri, Jan 19, 2018 at 12:44 PM, Markus Winter <[hidden email]> wrote:
> Why is ConsoleLogFilter not supported in pipeline.

Because…it isn’t yet. And JENKINS-38381, the better approach, was
blocked for the moment due to lack of test infrastructure.

> Timestamper plugin is also using a ConsoleLogFilter to add the timestamps in
> pipeline

That is using _contextual_ `ConsoleLogFilter`, which is definitely
supported. Just not `ConsoleLogFilter` as an _extension_.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3rBPXJVKRH4okGLuA%3Dbo4JEYj%2B-Csn9ujLQT6Fdjk2yw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: ConsoleLogFilter not working for builds on slaves

Tim Webster
Could I use 'contextual' ConsoleLogFilter to do what I need then?  I confess I don't really know what that means though - I'll look at the Timestamper plugin.

Although if JENKINS-38381 is going to be the preferred approach, I can also wait for that - or do both.  This is an internal project at the moment and I'm in no rush to release anything into the wild.

 

On Friday, 19 January 2018 18:17:51 UTC, Jesse Glick wrote:
On Fri, Jan 19, 2018 at 12:44 PM, Markus Winter <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="iG9Le7osBAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">m_wi...@...> wrote:
> Why is ConsoleLogFilter not supported in pipeline.

Because…it isn’t yet. And JENKINS-38381, the better approach, was
blocked for the moment due to lack of test infrastructure.

> Timestamper plugin is also using a ConsoleLogFilter to add the timestamps in
> pipeline

That is using _contextual_ `ConsoleLogFilter`, which is definitely
supported. Just not `ConsoleLogFilter` as an _extension_.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/0d3100d1-20a9-4ea1-bef4-fab52d975e43%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: ConsoleLogFilter not working for builds on slaves

Jesse Glick-4
On Fri, Jan 19, 2018 at 2:06 PM, Tim Webster <[hidden email]> wrote:
> Could I use 'contextual' ConsoleLogFilter to do what I need then?

If what you need is simply to *copy* all log lines somewhere else,
performance be damned, and you do not mind this being configured on a
per-job basis, then yes you could just make a `SimpleBuildWrapper`
defining a `ConsoleLogFilter`, which would work from either freestyle
projects (with a “build environment”) or Pipeline (with a block-scoped
step).

JENKINS-38381 would define a different API allowing the entire build
log to be sent to an external sink as the authoritative record,
meaning

· `$JENKINS_HOME/jobs/*/builds/*/log` need not even exist (except
perhaps as a cache)
· text produced by external processes launched on agents can be
streamed directly from there, never going over the Remoting channel to
the master

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr0ZJ4dmmS9KcOBXSaAvs7Rcx1o_2tVSUAOv6Y2hc%3Dh%3Dpw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: ConsoleLogFilter not working for builds on slaves

Tim Webster
OK - thanks for the info.  JENKINS-38381 definitely sounds like the way forward - keeping an eye on that..

On Friday, 19 January 2018 20:05:56 UTC, Jesse Glick wrote:
On Fri, Jan 19, 2018 at 2:06 PM, Tim Webster <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="W49tkhAzBAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">tim.w...@...> wrote:
> Could I use 'contextual' ConsoleLogFilter to do what I need then?

If what you need is simply to *copy* all log lines somewhere else,
performance be damned, and you do not mind this being configured on a
per-job basis, then yes you could just make a `SimpleBuildWrapper`
defining a `ConsoleLogFilter`, which would work from either freestyle
projects (with a “build environment”) or Pipeline (with a block-scoped
step).

JENKINS-38381 would define a different API allowing the entire build
log to be sent to an external sink as the authoritative record,
meaning

· `$JENKINS_HOME/jobs/*/builds/*/log` need not even exist (except
perhaps as a cache)
· text produced by external processes launched on agents can be
streamed directly from there, never going over the Remoting channel to
the master

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/8538f9c5-5ea3-4419-94ae-9af0c2a08b8f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.