Has anyone implemented OpenTelemetry or similar observability APIs in Jenkins?

[Matt Sicker]

Site: https://opentelemetry.io/

This seems like it could be a useful thing to integrate with,
particularly for Remoting-related network calls, but it could
potentially be extended to other HTTP client type calls and such for
richer tracing data. As a Jenkins admin, you'd be able to identify
issues in your Jenkins cluster more easily. Implemented properly, it
could even provide a way for users to submit traces to help debug
issues they find.

I looked at their Java library, and it seems to have an absurd amount
of dependencies compared to a proper logging library (which has no
required dependencies), but the gist of attaching traces and spans to
requests/responses is fairly simple to implement.

--
Matt Sicker
Senior Software Engineer, CloudBees

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAEot4ow2XtHJZh1gQ1DqZxwBYDRFtbv8Yi0KtPECt0beF8wxew%40mail.gmail.com.

[Cyrille Le Clerc-3]

Hello Matt,

For Your Information

• I'm writing a PoC to instrument Jenkins with OpenTelemetry.
• code: https://github.com/cyrille-leclerc/opentelemetry-plugin
• My focus for the moment is tracing the execution of jobs with OpenTelemetry distributed traces
• I have a glitch for which I am asking for help on https://groups.google.com/g/jenkinsci-dev/c/TEMMId7vzh0
• I plan later down the road to instrument:
• Jobs with OpenTelemetry metrics and logs
• Jenkins internals with OpenTelemetry metrics and logs
• We could also instrument Jenkins HTTP requests as you said

As OpenTelemetry is emerging as the standard for Observability / Monitoring with integrations with most observability technologies, open source and commercial, such as Jaeger, Elastic, Prometheus...), this integration will enable CI monitoring to most Jenkins users.

Cyrille

On Thursday, May 7, 2020 at 12:11:37 AM UTC+2, Matt Sicker wrote:

Site: <a href="https://opentelemetry.io/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fopentelemetry.io%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFA1S6djTp_Bfdt_siFvz7gcCn29A&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fopentelemetry.io%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFA1S6djTp_Bfdt_siFvz7gcCn29A&#39;;return true;">https://opentelemetry.io/

This seems like it could be a useful thing to integrate with,
particularly for Remoting-related network calls, but it could
potentially be extended to other HTTP client type calls and such for
richer tracing data. As a Jenkins admin, you'd be able to identify
issues in your Jenkins cluster more easily. Implemented properly, it
could even provide a way for users to submit traces to help debug
issues they find.

I looked at their Java library, and it seems to have an absurd amount
of dependencies compared to a proper logging library (which has no
required dependencies), but the gist of attaching traces and spans to
requests/responses is fairly simple to implement.

--
Matt Sicker
Senior Software Engineer, CloudBees

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/296be7c7-f98b-4f80-a3a8-f023ffb118c7o%40googlegroups.com.

[Matt Sicker]

Hey, great to see you again, Cyrille!

This sounds interesting. I'm still not a fan of their enormous
dependency stack, but I'm sure it's sufficient for prototyping. :)

On Tue, Jan 12, 2021 at 2:52 AM 'Cyrille Le Clerc' via Jenkins
Developers <[hidden email]> wrote:

>
> Hello Matt,
>
> For Your Information
>
> I'm writing a PoC to instrument Jenkins with OpenTelemetry.
>
> code: https://github.com/cyrille-leclerc/opentelemetry-plugin
>
> My focus for the moment is tracing the execution of jobs with OpenTelemetry distributed traces
>
> I have a glitch for which I am asking for help on https://groups.google.com/g/jenkinsci-dev/c/TEMMId7vzh0
>
> I plan later down the road to instrument:
>
> Jobs with OpenTelemetry metrics and logs
> Jenkins internals with OpenTelemetry metrics and logs
>
> We could also instrument Jenkins HTTP requests as you said
>
> As OpenTelemetry is emerging as the standard for Observability / Monitoring with integrations with most observability technologies, open source and commercial, such as Jaeger, Elastic, Prometheus...), this integration will enable CI monitoring to most Jenkins users.
>
> Cyrille
>
> On Thursday, May 7, 2020 at 12:11:37 AM UTC+2, Matt Sicker wrote:
>>
>> Site: https://opentelemetry.io/
>>
>> This seems like it could be a useful thing to integrate with,
>> particularly for Remoting-related network calls, but it could
>> potentially be extended to other HTTP client type calls and such for
>> richer tracing data. As a Jenkins admin, you'd be able to identify
>> issues in your Jenkins cluster more easily. Implemented properly, it
>> could even provide a way for users to submit traces to help debug
>> issues they find.
>>
>> I looked at their Java library, and it seems to have an absurd amount
>> of dependencies compared to a proper logging library (which has no
>> required dependencies), but the gist of attaching traces and spans to
>> requests/responses is fairly simple to implement.
>>
>> --
>> Matt Sicker
>> Senior Software Engineer, CloudBees
>
> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/296be7c7-f98b-4f80-a3a8-f023ffb118c7o%40googlegroups.com.

--
Matt Sicker
Senior Software Engineer, CloudBees

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAEot4oxM1tjmFjd_7ht89vzXiq0kq6B0C%2BveAzGR7O%2Be%3DQtxuw%40mail.gmail.com.

[Cyrille Le Clerc-3]

Thanks Matt,

> This sounds interesting. I'm still not a fan of their enormous dependency stack, but I'm sure it's sufficient for prototyping. :)

I understand your concern. I think that the Return on Investment remains very positive enabling integration with a growing number of Observability systems:

• Store metrics in the Time Series DB you want capturing all the correlation attributes to report on the dimension that matters to you: Prometheus, Elastic...
• Capture job executions as distributed traces exported in your prefered free or commercial distributed tracing system to have great visibility on the job executions correlated with the health of the underlying infrastructure and leveraging automated anomaly detection provided by your observability systems.
• Quality of the instrumentation as all the observability actors can join forces to focus on just one instrumentation per CI/CD system and hopefully some standard dashboard across all CI/CD systems rather than fragmenting efforts. When I looked at all the Jenkins plugins that report observability data to so many backends (InfluxDB, Graphite, Datadog, Elastic, syslog..), I felt the unification of the collector layer would be beneficial for everybody

Regarding the interference of GRPC dependencies, I'm confident that it will be painless for most use cases.

Cyrille

On Tuesday, January 26, 2021 at 6:09:20 PM UTC+1 [hidden email] wrote:

Hey, great to see you again, Cyrille!

This sounds interesting. I'm still not a fan of their enormous
dependency stack, but I'm sure it's sufficient for prototyping. :)

On Tue, Jan 12, 2021 at 2:52 AM 'Cyrille Le Clerc' via Jenkins
Developers <jenkin...@...> wrote:

>
> Hello Matt,
>
> For Your Information
>
> I'm writing a PoC to instrument Jenkins with OpenTelemetry.
>
> code: https://github.com/cyrille-leclerc/opentelemetry-plugin
>
> My focus for the moment is tracing the execution of jobs with OpenTelemetry distributed traces
>
> I have a glitch for which I am asking for help on https://groups.google.com/g/jenkinsci-dev/c/TEMMId7vzh0
>
> I plan later down the road to instrument:
>
> Jobs with OpenTelemetry metrics and logs
> Jenkins internals with OpenTelemetry metrics and logs
>
> We could also instrument Jenkins HTTP requests as you said
>
> As OpenTelemetry is emerging as the standard for Observability / Monitoring with integrations with most observability technologies, open source and commercial, such as Jaeger, Elastic, Prometheus...), this integration will enable CI monitoring to most Jenkins users.
>
> Cyrille
>
> On Thursday, May 7, 2020 at 12:11:37 AM UTC+2, Matt Sicker wrote:
>>
>> Site: https://opentelemetry.io/
>>
>> This seems like it could be a useful thing to integrate with,
>> particularly for Remoting-related network calls, but it could
>> potentially be extended to other HTTP client type calls and such for
>> richer tracing data. As a Jenkins admin, you'd be able to identify
>> issues in your Jenkins cluster more easily. Implemented properly, it
>> could even provide a way for users to submit traces to help debug
>> issues they find.
>>
>> I looked at their Java library, and it seems to have an absurd amount
>> of dependencies compared to a proper logging library (which has no
>> required dependencies), but the gist of attaching traces and spans to
>> requests/responses is fairly simple to implement.
>>
>> --
>> Matt Sicker
>> Senior Software Engineer, CloudBees
>
> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@....
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/296be7c7-f98b-4f80-a3a8-f023ffb118c7o%40googlegroups.com.

--
Matt Sicker
Senior Software Engineer, CloudBees

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/d22175f1-54b1-486b-a922-80c83e0bce20n%40googlegroups.com.

[Cyrille Le Clerc-3]

Hello,

We are happy to have published the first "GA version" of the OpenTelemetry plugin, a version 0.9: https://plugins.jenkins.io/opentelemetry/

Cyrille

On Wednesday, January 27, 2021 at 11:57:43 AM UTC+1 Cyrille Le Clerc wrote:

Thanks Matt,

> This sounds interesting. I'm still not a fan of their enormous dependency stack, but I'm sure it's sufficient for prototyping. :)

I understand your concern. I think that the Return on Investment remains very positive enabling integration with a growing number of Observability systems:

• Store metrics in the Time Series DB you want capturing all the correlation attributes to report on the dimension that matters to you: Prometheus, Elastic...
• Capture job executions as distributed traces exported in your prefered free or commercial distributed tracing system to have great visibility on the job executions correlated with the health of the underlying infrastructure and leveraging automated anomaly detection provided by your observability systems.
• Quality of the instrumentation as all the observability actors can join forces to focus on just one instrumentation per CI/CD system and hopefully some standard dashboard across all CI/CD systems rather than fragmenting efforts. When I looked at all the Jenkins plugins that report observability data to so many backends (InfluxDB, Graphite, Datadog, Elastic, syslog..), I felt the unification of the collector layer would be beneficial for everybody

Regarding the interference of GRPC dependencies, I'm confident that it will be painless for most use cases.

Cyrille

On Tuesday, January 26, 2021 at 6:09:20 PM UTC+1 [hidden email] wrote:

Hey, great to see you again, Cyrille!

This sounds interesting. I'm still not a fan of their enormous
dependency stack, but I'm sure it's sufficient for prototyping. :)

On Tue, Jan 12, 2021 at 2:52 AM 'Cyrille Le Clerc' via Jenkins
Developers <[hidden email]> wrote:

>
> Hello Matt,
>
> For Your Information
>
> I'm writing a PoC to instrument Jenkins with OpenTelemetry.
>
> code: https://github.com/cyrille-leclerc/opentelemetry-plugin
>
> My focus for the moment is tracing the execution of jobs with OpenTelemetry distributed traces
>
> I have a glitch for which I am asking for help on https://groups.google.com/g/jenkinsci-dev/c/TEMMId7vzh0
>
> I plan later down the road to instrument:
>
> Jobs with OpenTelemetry metrics and logs
> Jenkins internals with OpenTelemetry metrics and logs
>
> We could also instrument Jenkins HTTP requests as you said
>
> As OpenTelemetry is emerging as the standard for Observability / Monitoring with integrations with most observability technologies, open source and commercial, such as Jaeger, Elastic, Prometheus...), this integration will enable CI monitoring to most Jenkins users.
>
> Cyrille
>
> On Thursday, May 7, 2020 at 12:11:37 AM UTC+2, Matt Sicker wrote:
>>
>> Site: https://opentelemetry.io/
>>
>> This seems like it could be a useful thing to integrate with,
>> particularly for Remoting-related network calls, but it could
>> potentially be extended to other HTTP client type calls and such for
>> richer tracing data. As a Jenkins admin, you'd be able to identify
>> issues in your Jenkins cluster more easily. Implemented properly, it
>> could even provide a way for users to submit traces to help debug
>> issues they find.
>>
>> I looked at their Java library, and it seems to have an absurd amount
>> of dependencies compared to a proper logging library (which has no
>> required dependencies), but the gist of attaching traces and spans to
>> requests/responses is fairly simple to implement.
>>
>> --
>> Matt Sicker
>> Senior Software Engineer, CloudBees
>
> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/296be7c7-f98b-4f80-a3a8-f023ffb118c7o%40googlegroups.com.

--
Matt Sicker
Senior Software Engineer, CloudBees

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/876a62aa-66a8-4f4b-a396-e88b6373791an%40googlegroups.com.