[GSoc 2018] [Student Introduction] Jenkins Remoting over Message Bus/Queue

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[GSoc 2018] [Student Introduction] Jenkins Remoting over Message Bus/Queue

Pham Vu Tuan
Hi,

My name is Pham Vu Tuan. I am currently studying my final year in Computer Science at Nanyang Technological University from Singapore. I am going to graduate in May and looking to do some coding activities to get busy this summer. I have some experiences as a back-end developer through my interships before. My familiar programming languages are Java, Python and Go.

I am familiar with Jenkins as a CI server in my work, I hope I can contribute some to Jenkins. Currenty, I am quite interested in project "Jenkins Remoting over Message Bus/Queue" and want to understand more about the project. I think I can learn some interesting things by doing this project. I am a user of RabbitMQ in my recent company, so I have a basic knowledge about this queue messaging technology. Currently, I haven't had any idea/proposal yet, hope it will come along the way with the support of our Jenkins community.

I can be found on LinkedIn: https://www.linkedin.com/in/phamvutuan/ and Github: https://github.com/pvtuan10.

I am looking forward to having some contributions to Jenkins community.

Regards,
Pham Vu Tuan

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/ec445ea4-0567-4e68-b7ad-274ceea24956%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [GSoc 2018] [Student Introduction] Jenkins Remoting over Message Bus/Queue

Jesse Glick-4
On Tue, Feb 13, 2018 at 9:49 AM, Pham Vu Tuan <[hidden email]> wrote:
> I am quite interested in project
> "Jenkins Remoting over Message Bus/Queue" and want to understand more about
> the project.

Probably Oleg will give you some depth but to get started
understanding the background, look at subclasses of:

https://github.com/jenkinsci/remoting/blob/master/src/main/java/hudson/remoting/CommandTransport.java

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr2893s8R_0MCehmfkB6hrz7_ng93fQVs3y4FNqQXMDN0g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [GSoc 2018] [Student Introduction] Jenkins Remoting over Message Bus/Queue

Oleg Nenashev
Thanks for the interest!
I will try to provide a detailed writeup about the project.

Generally, the idea is to replace current communication layer by a pure message bus implementation. Below you can find some things from the top of my head:
  • All communications between master and agent happen over the message bus
    • Jenkins agents and masters can connect to a message bus.
      • The topology is TBD, it may be either embedded message bus (Jenkins master as a host) or an external service. The latter one is preferable IMHO
    • Jenkins agents query connection metadata over the bus.
      • Current implementations query metadata from an HTTP/HTTPs endpoint before connection. Getting rid of that would help
    • Command transport for message bus is implemented (as Jesse mentioned).
      • It would include transparent support of command invocation, RMI, classloading and data streaming
  • We utilize message bus features to make connections over Remoting more stable
    • Fault tolerance
      • The messaging layer survives short network outages
      • Nice2Have: We use multiple message bus hosts so they do not become a bottleneck
    • Traffic/Request prioritization
      • Currently we have an issue with logging/file transfer messaging overloading the channel and blocking system commands. We could use message bus features to prevent that
    • TBD: More efficient data streaming
  • Implementation:
    • Executable for the agent side
    • Jenkins plugin, which provides the master-side logic and serves the agent executable
  • We have a reference implementation for demos
    • E.g: a set of Docker images, which bring up a Jenkins cluster with a master and several agents (e.g. with Kubernetes or Docker Compose)
    • It may be somehow aligned with other ongoing activities (e.g. the demo could be built on the top of EverGreen distro)
  • TBD: a PoC for Master <=> Master communication
    • In Jenkins project we have some master => master plugins like Parameterized Remote Trigger. We could try to use a message bus to deliver messages between masters
  • TBD: Streaming agent logs to a log storage over the message bus
  • TBD: Remoting autoupdate over message bus
    • Message bust engine could be a bootstrap library, which loads Remoting library from the master after establishing the connection.
    • In Jenkins project we often see issues with update Remoting versions on the agent side. Although it would be great to solve the issue on the Remoting side, usage of such bootstrap could be a quick win
Best regards,
Oleg

вторник, 13 февраля 2018 г., 16:53:00 UTC+1 пользователь Jesse Glick написал:
On Tue, Feb 13, 2018 at 9:49 AM, Pham Vu Tuan <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="4N2vxgoXBAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">phamvu...@...> wrote:
> I am quite interested in project
> "Jenkins Remoting over Message Bus/Queue" and want to understand more about
> the project.

Probably Oleg will give you some depth but to get started
understanding the background, look at subclasses of:

<a href="https://github.com/jenkinsci/remoting/blob/master/src/main/java/hudson/remoting/CommandTransport.java" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Fremoting%2Fblob%2Fmaster%2Fsrc%2Fmain%2Fjava%2Fhudson%2Fremoting%2FCommandTransport.java\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGU0NzN9wmedPaHqm-1B8BvmcQrew&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Fremoting%2Fblob%2Fmaster%2Fsrc%2Fmain%2Fjava%2Fhudson%2Fremoting%2FCommandTransport.java\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGU0NzN9wmedPaHqm-1B8BvmcQrew&#39;;return true;">https://github.com/jenkinsci/remoting/blob/master/src/main/java/hudson/remoting/CommandTransport.java

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/23b3c653-bdab-4173-a043-ac270b5f7961%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [GSoc 2018] [Student Introduction] Jenkins Remoting over Message Bus/Queue

Pham Vu Tuan
Hi,

Thank you Oleg and Jesse for providing me useful details. I will try to get started and get back to you soon with my ideas.

Regards,
Pham Vu Tuan

On Wednesday, February 14, 2018 at 4:23:13 PM UTC+7, Oleg Nenashev wrote:
Thanks for the interest!
I will try to provide a detailed writeup about the project.

Generally, the idea is to replace current communication layer by a pure message bus implementation. Below you can find some things from the top of my head:
  • All communications between master and agent happen over the message bus
    • Jenkins agents and masters can connect to a message bus.
      • The topology is TBD, it may be either embedded message bus (Jenkins master as a host) or an external service. The latter one is preferable IMHO
    • Jenkins agents query connection metadata over the bus.
      • Current implementations query metadata from an HTTP/HTTPs endpoint before connection. Getting rid of that would help
    • Command transport for message bus is implemented (as Jesse mentioned).
      • It would include transparent support of command invocation, RMI, classloading and data streaming
  • We utilize message bus features to make connections over Remoting more stable
    • Fault tolerance
      • The messaging layer survives short network outages
      • Nice2Have: We use multiple message bus hosts so they do not become a bottleneck
    • Traffic/Request prioritization
      • Currently we have an issue with logging/file transfer messaging overloading the channel and blocking system commands. We could use message bus features to prevent that
    • TBD: More efficient data streaming
  • Implementation:
    • Executable for the agent side
    • Jenkins plugin, which provides the master-side logic and serves the agent executable
  • We have a reference implementation for demos
    • E.g: a set of Docker images, which bring up a Jenkins cluster with a master and several agents (e.g. with Kubernetes or Docker Compose)
    • It may be somehow aligned with other ongoing activities (e.g. the demo could be built on the top of <a href="https://github.com/jenkinsci/jep/tree/master/jep/300" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Fjep%2Ftree%2Fmaster%2Fjep%2F300\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEKa63NAiwqU8STMliOlgC9opwkfw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Fjep%2Ftree%2Fmaster%2Fjep%2F300\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEKa63NAiwqU8STMliOlgC9opwkfw&#39;;return true;">EverGreen distro)
  • TBD: a PoC for Master <=> Master communication
    • In Jenkins project we have some master => master plugins like Parameterized Remote Trigger. We could try to use a message bus to deliver messages between masters
  • TBD: Streaming agent logs to a log storage over the message bus
  • TBD: Remoting autoupdate over message bus
    • Message bust engine could be a bootstrap library, which loads Remoting library from the master after establishing the connection.
    • In Jenkins project we often see issues with update Remoting versions on the agent side. Although it would be great to solve the issue on the Remoting side, usage of such bootstrap could be a quick win
Best regards,
Oleg

вторник, 13 февраля 2018 г., 16:53:00 UTC+1 пользователь Jesse Glick написал:
On Tue, Feb 13, 2018 at 9:49 AM, Pham Vu Tuan <[hidden email]> wrote:
> I am quite interested in project
> "Jenkins Remoting over Message Bus/Queue" and want to understand more about
> the project.

Probably Oleg will give you some depth but to get started
understanding the background, look at subclasses of:

<a href="https://github.com/jenkinsci/remoting/blob/master/src/main/java/hudson/remoting/CommandTransport.java" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Fremoting%2Fblob%2Fmaster%2Fsrc%2Fmain%2Fjava%2Fhudson%2Fremoting%2FCommandTransport.java\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGU0NzN9wmedPaHqm-1B8BvmcQrew&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fjenkinsci%2Fremoting%2Fblob%2Fmaster%2Fsrc%2Fmain%2Fjava%2Fhudson%2Fremoting%2FCommandTransport.java\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGU0NzN9wmedPaHqm-1B8BvmcQrew&#39;;return true;">https://github.com/jenkinsci/remoting/blob/master/src/main/java/hudson/remoting/CommandTransport.java

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/394e00b8-5516-4e92-90c0-2fd5e01ab5ae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [GSoc 2018] [Student Introduction] Jenkins Remoting over Message Bus/Queue

Jesse Glick-4
In reply to this post by Oleg Nenashev
On Wed, Feb 14, 2018 at 4:23 AM, Oleg Nenashev <[hidden email]> wrote:
> Traffic/Request prioritization
>
> Currently we have an issue with logging/file transfer messaging overloading
> the channel and blocking system commands. We could use message bus features
> to prevent that

This could be a problem. The current API

https://github.com/jenkinsci/remoting/blob/563bff81f5639db0aad501f784a97e8af6742541/src/main/java/hudson/remoting/CommandTransport.java#L29-L33

specifies that packets must be delivered in order—a transport is not
permitted to rearrange them as an optimization.

I am not sure this needs fixing, anyway. Even large transfers over the
channel are generally broken up into chunks at a higher protocol
layer, permitting other `Command`s to be interleaved. And we are
independently working on architectural changes specifically to avoid
the two examples given—use of the Remoting channel to transfer large
files or significant volumes of log text.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1gq_3Hka3WUreUBq3%3DqWPSJG-BkfcTWrbnHYfiN%3Djs-w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [GSoc 2018] [Student Introduction] Jenkins Remoting over Message Bus/Queue

Gonçalo Garcia
In reply to this post by Pham Vu Tuan
Hello everyone,

My name is Gonçalo Garcia. I'm currently pursing a Master's Degree in Distributed Systems, and completed my Bachelor's in Computer Engineering last summer. I also have some internship experience in software development in one of the fastest growing start-ups in my country. I've mainly worked with Java, C# and to a lesser extent Python, C, and Bash.

This project caught my interest among all the GSOC proposals as it would allow me to gain some experience in distributed systems while learning and improving some very widely used technologies. Some of which I have worked directly with (RabbitMQ and other MQTT frameworks) and Jenkins which was part of the pipeline in the company where I interned. Who knows, it might even help me find a topic for my thesis.

I read Oleg's quite comprehensive overview of the project and it seems like a good starting point, but I'd also like to get familiar with the code base. Are there some tasks or bugs that need to be fixed that could be a good introduction? I'd like to get my hands "dirty" as soon as possible.

For more information about me, you can find me on Linkedin: https://www.linkedin.com/in/goncalotgarcia/ and Github: https://github.com/GoncaloGarcia (here you can get a better view of my open source contributions).


I'm eager to help, and hopefully we can keep improving Jenkins.

Best Regards,
Gonçalo Garcia


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/dc2b2be3-66a5-4990-928f-fb36d08fef69%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [GSoc 2018] [Student Introduction] Jenkins Remoting over Message Bus/Queue

Oleg Nenashev
Hi Gonçalo,

Thanks for the interest!

I read Oleg's quite comprehensive overview of the project and it seems like a good starting point, but I'd also like to get familiar with the code base. Are there some tasks or bugs that need to be fixed that could be a good introduction? I'd like to get my hands "dirty" as soon as possible.

Remoting is an internal engine, so it's hard to find a "newbie-friendly" ticket. I would suggest starting from establishing the Jenkins master with agents and trying various remote operations. OTOH it is still possible to find some low-hanging fruits, especially in the agent-related area.

I have created a JIRA filter for newbie-friendly issues specific to remoting. The filter is here (you would need to login to JIRA I'd guess). I will make sure to extend the list of tickets there over this week.

Best regards,
Oleg

On Saturday, February 17, 2018 at 7:46:52 PM UTC+1, Gonçalo Garcia wrote:
Hello everyone,

My name is Gonçalo Garcia. I'm currently pursing a Master's Degree in Distributed Systems, and completed my Bachelor's in Computer Engineering last summer. I also have some internship experience in software development in one of the fastest growing start-ups in my country. I've mainly worked with Java, C# and to a lesser extent Python, C, and Bash.

This project caught my interest among all the GSOC proposals as it would allow me to gain some experience in distributed systems while learning and improving some very widely used technologies. Some of which I have worked directly with (RabbitMQ and other MQTT frameworks) and Jenkins which was part of the pipeline in the company where I interned. Who knows, it might even help me find a topic for my thesis.

I read Oleg's quite comprehensive overview of the project and it seems like a good starting point, but I'd also like to get familiar with the code base. Are there some tasks or bugs that need to be fixed that could be a good introduction? I'd like to get my hands "dirty" as soon as possible.

For more information about me, you can find me on Linkedin: <a href="https://www.linkedin.com/in/goncalotgarcia/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.linkedin.com%2Fin%2Fgoncalotgarcia%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHO5yaM7y9UG8Mfn-p0bvWTM9IoOw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.linkedin.com%2Fin%2Fgoncalotgarcia%2F\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHO5yaM7y9UG8Mfn-p0bvWTM9IoOw&#39;;return true;">https://www.linkedin.com/in/goncalotgarcia/ and Github: <a href="https://github.com/GoncaloGarcia" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FGoncaloGarcia\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMaY3Go5KBBM-hi6c9XITpzJXfLw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FGoncaloGarcia\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEMaY3Go5KBBM-hi6c9XITpzJXfLw&#39;;return true;">https://github.com/GoncaloGarcia (here you can get a better view of my open source contributions).


I'm eager to help, and hopefully we can keep improving Jenkins.

Best Regards,
Gonçalo Garcia


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/e47cb02b-2723-4afc-8cbf-9af84ee45762%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.