Efficient class/jar prefetching in remoting

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Efficient class/jar prefetching in remoting

Kohsuke Kawaguchi
Administrator
(Context: see https://github.com/jenkinsci/remoting/pull/10)

I've got the new code working under the Maven job type to see the effect of prefetching. Here is the summary of classloader activities in building https://svn.jenkins-ci.org/trunk/jenkins/test-projects/model-maven-project/

Class loading count=801
Class loading prefetch hit=372 (46%)
Resource loading count=11

The new code manages to avoid sending individual class/resource file images on the wire completely, and they are instead all retrieved from locally cached jar files.

The prefetch hit ratio 46% means we were able to cut the number of roundtrips to 54% of what it was before. Interestingly, this 46% number is very consistent across different call patterns --- the slave itself had 48% prefetch hit ratio.

I haven't measured the difference in the number of bytes transferred.

I wonder what can be done to further improve the prefetch hit ratio.

The complete call sequence details at https://gist.github.com/kohsuke/5561414

--
Kohsuke Kawaguchi

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Efficient class/jar prefetching in remoting

Dean Yu-3
Do you also prefetch inherited classes? I wonder if that would help. Also, can you measure the number of classes that were prefetched but never used?

  -- Dean

On Saturday, May 11, 2013 2:10:18 PM UTC-7, Kohsuke Kawaguchi wrote:

> (Context: see https://github.com/jenkinsci/remoting/pull/10)
>
>
> I've got the new code working under the Maven job type to see the effect of prefetching. Here is the summary of classloader activities in building https://svn.jenkins-ci.org/trunk/jenkins/test-projects/model-maven-project/
>
>
>
> Class loading count=801
> Class loading prefetch hit=372 (46%)
> Resource loading count=11
>
>
>
> The new code manages to avoid sending individual class/resource file images on the wire completely, and they are instead all retrieved from locally cached jar files.
>
>
>
> The prefetch hit ratio 46% means we were able to cut the number of roundtrips to 54% of what it was before. Interestingly, this 46% number is very consistent across different call patterns --- the slave itself had 48% prefetch hit ratio.
>
>
>
> I haven't measured the difference in the number of bytes transferred.
>
>
> I wonder what can be done to further improve the prefetch hit ratio.
>
>
>
> The complete call sequence details at https://gist.github.com/kohsuke/5561414
>
>
> --
> Kohsuke Kawaguchi

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: Efficient class/jar prefetching in remoting

Dean Yu-3
I've been noodling over this, and I'm not convinced that optimistic
prefetching of classes is going to lead to an overall win. In the end this
is about latency, and the hypothesis is that you save on the latency of
round trip calls for loading a single class at a time by sending many
classes up front. The only way this would be true is if the amount of
latency you add by increasing the size of the original payload is less
than the sum of the latency for each round trip call that is saved. You
also reduce the amount you could save if you wind up sending more classes
than is actually needed as part of the remote calling chain. I find it
hard to believe that you can achieve the necessary savings, at lease
purely from the remoting layer. It seems that you'd need some code in
Jenkins to help tune the prefetch algorithm.

  -- Dean

On 5/12/13 8:08 AM, "Dean Yu" <[hidden email]> wrote:

>Do you also prefetch inherited classes? I wonder if that would help.
>Also, can you measure the number of classes that were prefetched but
>never used?
>
>  -- Dean
>
>On Saturday, May 11, 2013 2:10:18 PM UTC-7, Kohsuke Kawaguchi wrote:
>> (Context: see https://github.com/jenkinsci/remoting/pull/10)
>>
>>
>> I've got the new code working under the Maven job type to see the
>>effect of prefetching. Here is the summary of classloader activities in
>>building
>>https://svn.jenkins-ci.org/trunk/jenkins/test-projects/model-maven-projec
>>t/
>>
>>
>>
>> Class loading count=801
>> Class loading prefetch hit=372 (46%)
>> Resource loading count=11
>>
>>
>>
>> The new code manages to avoid sending individual class/resource file
>>images on the wire completely, and they are instead all retrieved from
>>locally cached jar files.
>>
>>
>>
>> The prefetch hit ratio 46% means we were able to cut the number of
>>roundtrips to 54% of what it was before. Interestingly, this 46% number
>>is very consistent across different call patterns --- the slave itself
>>had 48% prefetch hit ratio.
>>
>>
>>
>> I haven't measured the difference in the number of bytes transferred.
>>
>>
>> I wonder what can be done to further improve the prefetch hit ratio.
>>
>>
>>
>> The complete call sequence details at
>>https://gist.github.com/kohsuke/5561414
>>
>>
>> --
>> Kohsuke Kawaguchi
>
>--
>You received this message because you are subscribed to the Google Groups
>"Jenkins Developers" group.
>To unsubscribe from this group and stop receiving emails from it, send an
>email to [hidden email].
>For more options, visit https://groups.google.com/groups/opt_out.
>
>


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: Efficient class/jar prefetching in remoting

kohsuke Kawaguchi (CB)
In reply to this post by Dean Yu-3
On 05/12/2013 08:08 AM, Dean Yu wrote:
> Do you also prefetch inherited classes?

No. The prefetching is microscopic, in that it currently looks for
classes that are referenced from the class being requested (in the hope
that those referenced classes are likely needed soon.)

> Also, can you measure the number of classes that were prefetched but never used?

Right, ideally I'd produce this in a format that allows some reasoning.
Let's try that.


>    -- Dean
>
> On Saturday, May 11, 2013 2:10:18 PM UTC-7, Kohsuke Kawaguchi wrote:
>> (Context: see https://github.com/jenkinsci/remoting/pull/10)
>>
>>
>> I've got the new code working under the Maven job type to see the effect of prefetching. Here is the summary of classloader activities in building https://svn.jenkins-ci.org/trunk/jenkins/test-projects/model-maven-project/
>>
>>
>>
>> Class loading count=801
>> Class loading prefetch hit=372 (46%)
>> Resource loading count=11
>>
>>
>>
>> The new code manages to avoid sending individual class/resource file images on the wire completely, and they are instead all retrieved from locally cached jar files.
>>
>>
>>
>> The prefetch hit ratio 46% means we were able to cut the number of roundtrips to 54% of what it was before. Interestingly, this 46% number is very consistent across different call patterns --- the slave itself had 48% prefetch hit ratio.
>>
>>
>>
>> I haven't measured the difference in the number of bytes transferred.
>>
>>
>> I wonder what can be done to further improve the prefetch hit ratio.
>>
>>
>>
>> The complete call sequence details at https://gist.github.com/kohsuke/5561414
>>
>>
>> --
>> Kohsuke Kawaguchi
>


--
Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/
Try Jenkins Enterprise, our professional version of Jenkins

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: Efficient class/jar prefetching in remoting

Jesse Glick-4
In reply to this post by Dean Yu-3
On 05/13/2013 08:11 PM, Dean Yu wrote:
> the hypothesis is that you save on the latency of
> round trip calls for loading a single class at a time by sending many
> classes up front. The only way this would be true is if the amount of
> latency you add by increasing the size of the original payload is less
> than the sum of the latency for each round trip call that is saved.

Right, but I hope this is satisfied by most of the payloads being small enough to fit into one transport packet even considering prefetch.

The proposed patch is a little tricky. My original work on the branch, which seemed to provide some benefit, only dealt with class prefetch: before the master sends a
class file to the slave, it scans the bytecode for class constants referring to to related classes (dependencies, inner classes, etc.), determines which class loader
would be used for those if the code ever asked for them to be loaded, and bundles all this up in the response so that the slave can quickly load a series of classes
without making a new round trip.

We also wanted to deliver entire JAR files, both to try to push as much actual data in a single burst as possible, and to allow the slave to cache JAR files after an
agent restart. The tricky part is that the slave does not know which JAR file it is supposed to consult for a given class name, since the remoting layer models an
arbitrary class loader hierarchy (or graph!) rather than a “flat classpath”. So the master still needs to inform the slave of the code source for each proposed class
load. It can however avoid a round trip per class loader by using the original prefetch heuristic to predict what classes might soon be loaded as well.

The result would be that when first setting up a slave (or after updating Jenkins and/or plugins) you would get some big packets full of JAR content being pushed down the
pipe at full speed, which does not need to wait for the slave to respond and so can be relatively efficient if the transport itself is (and Kohsuke did some fixes here in
1.509). Then when taking a slave online, or starting to use some complex plugins for the first time in a given slave connection, the class loading will require some
round-trip communication to inform the slave of where to load everything from (even though it already has the bytecode on disk); but most of these little packets will
include information about several classes rather than just one, so we reduce the overall latency.

Another option initially considered, but not implemented in the branch, was to avoid all of the round-trip activity by having the master send a serializable proxy which
would effect the same class loading logic as exists on the master. For example, a model for the loader of a Jenkins plugin would refer to the models of the Jenkins core
and any plugin dependencies, with some additional fields for options such as plugin-first class loading. This could be very efficient but for it to work requires that
every ClassLoader in the master which loads anything serialized to the slave also supply a proxy, meaning more code in Jenkins core to troubleshoot and maintain.

The above is my interpretation of what the branch currently does, so Kohsuke please correct me if I have got it wrong!

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.


Reply | Threaded
Open this post in threaded view
|

Re: Efficient class/jar prefetching in remoting

Kohsuke Kawaguchi
Administrator
In reply to this post by Dean Yu-3
First, to be really on the same page, it might be useful to think of this change as a composition of two changes that are related.

The first one is the jar file caching on the other side of the channel. When one side wants to load class "Foo" from the other side, it used to be that we sent the class file image in the response. With this branch, the response can be "that class is in the jar file e11f8a3c1ee73e09b19e344f9fe99cb5" (which refers to the check sum of the jar file). The receiver, once it gets this response, can either look up this jar file in its local cache, or make a separate call to retrieve this jar file.

So the response to the classloading request is actualy getting a lot smaller, because now we are mostly just sending around the checksum of a jar file, instead of the actual class file.


The second part of what this branch does is prefetching. When one side wants to load "Foo" from the other side, we not only tell them that Foo is in the jar file e11f8a3c1ee73e09b19e344f9fe99cb5, but we'll also tell them that Bar is from the same jar file, and Zot is from the jar file cb58012189e5b726c347219d6d5c73c8.

This eats up some of the saving we achieved in the first change, but the response should still come in far smaller than what we had before.



And now that we are on the same page, I think you are mixing up latency and bandwidth. As you noted, this is an optimization for high latency network, and I still assume there's fair bandwidth. Adding to the size of the request/response is not too bad in such a network, because the time it takes to do so is a function of bandwidth. For example, even on 1MB/sec network you can get 100KB sent for 100ms.

In contrast, the delay induced by a roundtrip is a function of latency, so if your roundtrip time is 100ms, saving single roundtrip wins you 100ms.

Thus in this hypothetical network, saving one roundtrip by adding 50KB to the response is a win, and I think this branch is a net win for this kind of situation.


This branch hurts the initial start-up cost of a new slave, but for most situations I think this is a net-win.




2013/5/13 Dean Yu <[hidden email]>
I've been noodling over this, and I'm not convinced that optimistic
prefetching of classes is going to lead to an overall win. In the end this
is about latency, and the hypothesis is that you save on the latency of
round trip calls for loading a single class at a time by sending many
classes up front. The only way this would be true is if the amount of
latency you add by increasing the size of the original payload is less
than the sum of the latency for each round trip call that is saved. You
also reduce the amount you could save if you wind up sending more classes
than is actually needed as part of the remote calling chain. I find it
hard to believe that you can achieve the necessary savings, at lease
purely from the remoting layer. It seems that you'd need some code in
Jenkins to help tune the prefetch algorithm.

  -- Dean

On 5/12/13 8:08 AM, "Dean Yu" <[hidden email]> wrote:

>Do you also prefetch inherited classes? I wonder if that would help.
>Also, can you measure the number of classes that were prefetched but
>never used?
>
>  -- Dean
>
>On Saturday, May 11, 2013 2:10:18 PM UTC-7, Kohsuke Kawaguchi wrote:
>> (Context: see https://github.com/jenkinsci/remoting/pull/10)
>>
>>
>> I've got the new code working under the Maven job type to see the
>>effect of prefetching. Here is the summary of classloader activities in
>>building
>>https://svn.jenkins-ci.org/trunk/jenkins/test-projects/model-maven-projec
>>t/
>>
>>
>>
>> Class loading count=801
>> Class loading prefetch hit=372 (46%)
>> Resource loading count=11
>>
>>
>>
>> The new code manages to avoid sending individual class/resource file
>>images on the wire completely, and they are instead all retrieved from
>>locally cached jar files.
>>
>>
>>
>> The prefetch hit ratio 46% means we were able to cut the number of
>>roundtrips to 54% of what it was before. Interestingly, this 46% number
>>is very consistent across different call patterns --- the slave itself
>>had 48% prefetch hit ratio.
>>
>>
>>
>> I haven't measured the difference in the number of bytes transferred.
>>
>>
>> I wonder what can be done to further improve the prefetch hit ratio.
>>
>>
>>
>> The complete call sequence details at
>>https://gist.github.com/kohsuke/5561414
>>
>>
>> --
>> Kohsuke Kawaguchi
>
>--
>You received this message because you are subscribed to the Google Groups
>"Jenkins Developers" group.
>To unsubscribe from this group and stop receiving emails from it, send an
>email to [hidden email].
>For more options, visit https://groups.google.com/groups/opt_out.
>
>


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.





--
Kohsuke Kawaguchi

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Efficient class/jar prefetching in remoting

Kohsuke Kawaguchi
Administrator
In reply to this post by Jesse Glick-4
Looking at the list of classloading sequence and how prefetch is taking effect, one thing I noticed is that when JVM loads Foo, it immediately resolves all the interfaces and super classes that Foo extends/implements. This makes sense, but our prefetching logic doesn't take this into account.

So when you have a deeply nested type hierarchy, this results in multiple roundtrips. A related inefficiency applies in the following situation:

- French side requests Foo
- British side sends Foo and prefetches Bar
- ...
- French side tries to load Bar, finds it prefetched
- French side requests the base interfaces/super classes for Bar, resulting in multiple roundtrips.

Finding super classes and interfaces require a deeper parsing of a class file, though. 


2013/5/14 Jesse Glick <[hidden email]>
On 05/13/2013 08:11 PM, Dean Yu wrote:
the hypothesis is that you save on the latency of
round trip calls for loading a single class at a time by sending many
classes up front. The only way this would be true is if the amount of
latency you add by increasing the size of the original payload is less
than the sum of the latency for each round trip call that is saved.

Right, but I hope this is satisfied by most of the payloads being small enough to fit into one transport packet even considering prefetch.

The proposed patch is a little tricky. My original work on the branch, which seemed to provide some benefit, only dealt with class prefetch: before the master sends a class file to the slave, it scans the bytecode for class constants referring to to related classes (dependencies, inner classes, etc.), determines which class loader would be used for those if the code ever asked for them to be loaded, and bundles all this up in the response so that the slave can quickly load a series of classes without making a new round trip.

We also wanted to deliver entire JAR files, both to try to push as much actual data in a single burst as possible, and to allow the slave to cache JAR files after an agent restart. The tricky part is that the slave does not know which JAR file it is supposed to consult for a given class name, since the remoting layer models an arbitrary class loader hierarchy (or graph!) rather than a “flat classpath”. So the master still needs to inform the slave of the code source for each proposed class load. It can however avoid a round trip per class loader by using the original prefetch heuristic to predict what classes might soon be loaded as well.

The result would be that when first setting up a slave (or after updating Jenkins and/or plugins) you would get some big packets full of JAR content being pushed down the pipe at full speed, which does not need to wait for the slave to respond and so can be relatively efficient if the transport itself is (and Kohsuke did some fixes here in 1.509). Then when taking a slave online, or starting to use some complex plugins for the first time in a given slave connection, the class loading will require some round-trip communication to inform the slave of where to load everything from (even though it already has the bytecode on disk); but most of these little packets will include information about several classes rather than just one, so we reduce the overall latency.

Another option initially considered, but not implemented in the branch, was to avoid all of the round-trip activity by having the master send a serializable proxy which would effect the same class loading logic as exists on the master. For example, a model for the loader of a Jenkins plugin would refer to the models of the Jenkins core and any plugin dependencies, with some additional fields for options such as plugin-first class loading. This could be very efficient but for it to work requires that every ClassLoader in the master which loads anything serialized to the slave also supply a proxy, meaning more code in Jenkins core to troubleshoot and maintain.

The above is my interpretation of what the branch currently does, so Kohsuke please correct me if I have got it wrong!


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.





--
Kohsuke Kawaguchi

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Efficient class/jar prefetching in remoting

Jesse Glick-4
On 05/15/2013 01:53 AM, Kohsuke Kawaguchi wrote:
> when JVM loads Foo, it immediately resolves all the interfaces and super classes that Foo extends/implements. This makes sense, but our prefetching logic doesn't take this into account.

No? File at

https://github.com/jenkinsci/constant-pool-scanner/issues

please. The unit tests make it pretty easy to fix that sort of thing.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.