copying artifacts from slave is slow

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

copying artifacts from slave is slow

Rex Chan
Hi all,

I've recently setup a hudson slave (starte by connecting via ssh) to run
a few projects. Building is fine except for the final step - copying
artifacts from slave to master. Both master and slave are on the same
local network, on a 100 Mb switch and it takes over 10 min to copy to
10Mb files.

The master is a FreeBSD box and the slave is an PowerPC mac.

I'm curious to know if anybody else is experiencing this with copying
artifacts from slave nodes?

--

Rex Chan
-------------------------->  
ish
http://www.ish.com.au
Level 1, 30 Wilson Street Newtown 2042 Australia
phone +61 2 9550 5001   fax +61 2 9550 4001

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: copying artifacts from slave is slow

Kohsuke Kawaguchi
Administrator

In the past we had a few buffering related bugs that made copying slow,
but AFAIK all such issues are fixed now.

One thing that might help us is for you to run a thread dump on the
slave (http://server/hudson/computer/SLAVENAME/threadDump) during the
slow copy is happening. Do this several times and send the output to us.

Also, it would be mighty helpful if you can use a packet monitor like
Wireshark to capture the network traffic, to see if you see a lot of
small packets.

Rex Chan wrote:

> Hi all,
>
> I've recently setup a hudson slave (starte by connecting via ssh) to run
> a few projects. Building is fine except for the final step - copying
> artifacts from slave to master. Both master and slave are on the same
> local network, on a 100 Mb switch and it takes over 10 min to copy to
> 10Mb files.
>
> The master is a FreeBSD box and the slave is an PowerPC mac.
>
> I'm curious to know if anybody else is experiencing this with copying
> artifacts from slave nodes?
>

--
Kohsuke Kawaguchi
Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: copying artifacts from slave is slow

Rex Chan
Hi Kohsuke,

Kohsuke Kawaguchi <[hidden email]> wrote:

> One thing that might help us is for you to run a thread dump on the
> slave (http://server/hudson/computer/SLAVENAME/threadDump) during the
> slow copy is happening. Do this several times and send the output to
> us.

Did you mean http://server/hudson/computer/SLAVENAME/systeminfo ?
I tried with http://server/hudson/computer/SLAVENAME/threadDump but
tomcat gives me a 404 error.
 
> Also, it would be mighty helpful if you can use a packet monitor like
> Wireshark to capture the network traffic, to see if you see a lot of
> small packets.

I haven't had much experience with using wireshark, but what range of
sizes would be considered small?

--

Rex Chan
-------------------------->  
ish
http://www.ish.com.au
Level 1, 30 Wilson Street Newtown 2042 Australia
phone +61 2 9550 5001   fax +61 2 9550 4001

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: copying artifacts from slave is slow

Bruce Chapman
Rex,


Did you replace SLAVENAME (in the supplied URL below) with the actual name
of the slave? I think that was Kohsuke's intention, tho' if you are new to
this list, you might not have understood that.

Bruce

-----Original Message-----
From: Rex Chan [mailto:[hidden email]]
Sent: Tuesday, 14 April 2009 4:35 p.m.
To: [hidden email]
Subject: Re: copying artifacts from slave is slow

Hi Kohsuke,

Kohsuke Kawaguchi <[hidden email]> wrote:

> One thing that might help us is for you to run a thread dump on the
> slave (http://server/hudson/computer/SLAVENAME/threadDump) during the
> slow copy is happening. Do this several times and send the output to
> us.

Did you mean http://server/hudson/computer/SLAVENAME/systeminfo ?
I tried with http://server/hudson/computer/SLAVENAME/threadDump but
tomcat gives me a 404 error.
 
> Also, it would be mighty helpful if you can use a packet monitor like
> Wireshark to capture the network traffic, to see if you see a lot of
> small packets.

I haven't had much experience with using wireshark, but what range of
sizes would be considered small?

--

Rex Chan
-------------------------->  
ish
http://www.ish.com.au
Level 1, 30 Wilson Street Newtown 2042 Australia
phone +61 2 9550 5001   fax +61 2 9550 4001

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: copying artifacts from slave is slow

Rex Chan
In reply to this post by Rex Chan
On Tue, 14 Apr 2009 14:35:00 +1000
Rex Chan <[hidden email]> wrote:

> Did you mean http://server/hudson/computer/SLAVENAME/systeminfo ?

There's a thread dump section here in systeminfo:

Thread Dump
Ping thread for channel hudson.remoting.Channel@b82368:channel

java.lang.Thread.sleep(Native Method)
hudson.remoting.PingThread.run(PingThread.java:64)

Finalizer

java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

pool-1-thread-2

java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:474)
java.lang.UNIXProcess.waitFor(UNIXProcess.java:112)
hudson.Proc$LocalProc.join(Proc.java:147)
hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:504)
hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:485)
hudson.remoting.UserRequest.perform(UserRequest.java:97)
hudson.remoting.UserRequest.perform(UserRequest.java:46)
hudson.remoting.Request$2.run(Request.java:236)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
java.util.concurrent.FutureTask.run(FutureTask.java:123)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
java.lang.Thread.run(Thread.java:613)

main

java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:474)
hudson.remoting.Channel.join(Channel.java:568)
hudson.remoting.Launcher.main(Launcher.java:359)
hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:308)
hudson.remoting.Launcher.main(Launcher.java:187)

Channel reader thread: channel

java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2196)
java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2486)
java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2496)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1273)
java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
hudson.remoting.Channel$ReaderThread.run(Channel.java:665)

Reference Handler

java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:474)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)

pool-1-thread-8

java.lang.Thread.dumpThreads(Native Method)
java.lang.Thread.getAllStackTraces(Thread.java:1452)
hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:78)
hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:69)
hudson.remoting.UserRequest.perform(UserRequest.java:97)
hudson.remoting.UserRequest.perform(UserRequest.java:46)
hudson.remoting.Request$2.run(Request.java:236)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
java.util.concurrent.FutureTask.run(FutureTask.java:123)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
java.lang.Thread.run(Thread.java:613)

pool-1-thread-3

java.io.PrintStream.write(PrintStream.java:414)
java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1685)
java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1594)
java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1173)
java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1127)
java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1178)
java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1127)
java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1178)
java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1127)
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1284)
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
hudson.remoting.Channel.send(Channel.java:347)
hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:112)
hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:96)
java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:161)
java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:118)
java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:72)
java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
org.apache.tools.tar.TarBuffer.writeBlock(TarBuffer.java:410)
org.apache.tools.tar.TarBuffer.writeRecord(TarBuffer.java:388)
org.apache.tools.tar.TarOutputStream.write(TarOutputStream.java:332)
hudson.FilePath.writeToTar(FilePath.java:1166)
hudson.FilePath.access$600(FilePath.java:150)
hudson.FilePath$28.invoke(FilePath.java:1101)
hudson.FilePath$28.invoke(FilePath.java:1098)
hudson.FilePath$FileCallableWrapper.call(FilePath.java:1510)
hudson.remoting.UserRequest.perform(UserRequest.java:97)
hudson.remoting.UserRequest.perform(UserRequest.java:46)
hudson.remoting.Request$2.run(Request.java:236)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
java.util.concurrent.FutureTask.run(FutureTask.java:123)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
java.lang.Thread.run(Thread.java:613)

Signal Dispatcher

ant -file build.xml release-all: stdout copier

java.io.FileInputStream.readBytes(Native Method)
java.io.FileInputStream.read(FileInputStream.java:194)
java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
java.io.BufferedInputStream.read(BufferedInputStream.java:313)
java.io.FilterInputStream.read(FilterInputStream.java:90)
hudson.util.StreamCopyThread.run(StreamCopyThread.java:50)

process reaper

java.lang.UNIXProcess.waitForProcessExit(Native Method)
java.lang.UNIXProcess.access$700(UNIXProcess.java:17)
java.lang.UNIXProcess$2$1.run(UNIXProcess.java:83)

--

Rex Chan
-------------------------->  
ish
http://www.ish.com.au
Level 1, 30 Wilson Street Newtown 2042 Australia
phone +61 2 9550 5001   fax +61 2 9550 4001

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Re: copying artifacts from slave is slow

Salim, Fadhley (CALYON)
In reply to this post by Kohsuke Kawaguchi
Kohsuke,

Speaking of the time taken to copy artifacts, I've noticed that the time
taken to download them from the main Hudso GUI also seems very slow,
especially compared to downloading other files from the same tomcat
server. For example on one very modest server I can serve static files
at more than 1000 kbit/s. When I download a file from the Hudson GUI or
API I get under 200kbit/s.

Is it likely that your new fix will have solved this problem as well?

Thanks

-----Original Message-----
From: Kohsuke Kawaguchi [mailto:[hidden email]]
Sent: 14 April 2009 03:12
To: [hidden email]
Subject: Re: copying artifacts from slave is slow


In the past we had a few buffering related bugs that made copying slow,
but AFAIK all such issues are fixed now.

One thing that might help us is for you to run a thread dump on the
slave (http://server/hudson/computer/SLAVENAME/threadDump) during the
slow copy is happening. Do this several times and send the output to us.

Also, it would be mighty helpful if you can use a packet monitor like
Wireshark to capture the network traffic, to see if you see a lot of
small packets.

Rex Chan wrote:
> Hi all,
>
> I've recently setup a hudson slave (starte by connecting via ssh) to
> run a few projects. Building is fine except for the final step -
> copying artifacts from slave to master. Both master and slave are on
> the same local network, on a 100 Mb switch and it takes over 10 min to

> copy to 10Mb files.
>
> The master is a FreeBSD box and the slave is an PowerPC mac.
>
> I'm curious to know if anybody else is experiencing this with copying
> artifacts from slave nodes?
>


--
Kohsuke Kawaguchi
Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/

This email does not create a legal relationship between any member of the
Cr=E9dit Agricole group and the recipient or constitute investment advice.
The content of this email should not be copied or disclosed (in whole or
part) to any other person. It may contain information which is
confidential, privileged or otherwise protected from disclosure. If you
are not the intended recipient, you should notify us and delete it from
your system. Emails may be monitored, are not secure and may be amended,
destroyed or contain viruses and in communicating with us such conditions
are accepted. Any content which does not relate to business matters is not
endorsed by us.

Calyon is authorised by the Comit=e9 des Etablissements de Cr=e9dit et des
Entreprises d'Investissement (CECEI) and supervised by the Commission Bancaire
in France and subject to limited regulation by the Financial Services Authority.
Details about the extent of our regulation by the Financial Services Authority
are available from us on request. Calyon is incorporated in France with limited
liability and registered in England & Wales. Registration number: FC008194.
Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA.
This message and/or any  attachments is intended for the sole use of its addressee.
If you are not the addressee, please immediately notify the sender and then destroy the message.
As this message and/or any attachments may have been altered without our knowledge, its content is not legally binding on CALYON Crédit Agricole CIB.
All rights reserved.

Ce message et ses pièces jointes est destiné à l'usage exclusif de son destinataire.
Si vous recevez ce message par erreur, merci d'en aviser immédiatement l'expéditeur et de le détruire ensuite.
Le présent message pouvant être altéré à notre insu, CALYON Crédit Agricole CIB ne peut pas être engagé par son contenu.
Tous droits réservés.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: copying artifacts from slave is slow

Kohsuke Kawaguchi
Administrator
In reply to this post by Rex Chan
Rex Chan wrote:

> Hi Kohsuke,
>
> Kohsuke Kawaguchi <[hidden email]> wrote:
>
>> One thing that might help us is for you to run a thread dump on the
>> slave (http://server/hudson/computer/SLAVENAME/threadDump) during the
>> slow copy is happening. Do this several times and send the output to
>> us.
>
> Did you mean http://server/hudson/computer/SLAVENAME/systeminfo ?
> I tried with http://server/hudson/computer/SLAVENAME/threadDump but
> tomcat gives me a 404 error.
That's right, sorry about that.

>> Also, it would be mighty helpful if you can use a packet monitor like
>> Wireshark to capture the network traffic, to see if you see a lot of
>> small packets.
>
> I haven't had much experience with using wireshark, but what range of
> sizes would be considered small?

If the traffic is properly buffered, every packet is almost the full
size. In normal TCP/IP network that's around 1500 bytes.

OTOH, if things are horribly unbuffered, you'll see the TCP payload of a
few bytes, which should be about 50-60 bytes if you includ TCP/IP headers.

--
Kohsuke Kawaguchi
Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: copying artifacts from slave is slow

Kohsuke Kawaguchi
Administrator
In reply to this post by Salim, Fadhley (CALYON)
Fadhley Salim wrote:

> Kohsuke,
>
> Speaking of the time taken to copy artifacts, I've noticed that the time
> taken to download them from the main Hudso GUI also seems very slow,
> especially compared to downloading other files from the same tomcat
> server. For example on one very modest server I can serve static files
> at more than 1000 kbit/s. When I download a file from the Hudson GUI or
> API I get under 200kbit/s.
>
> Is it likely that your new fix will have solved this problem as well?
No, those two issues sounds very different issues.

Again, the same requests to you --- do the download, and get the thread
dump (http://server/hudson/threadDump) while the download is in
progress. And if you can, please run the packet sniffer so that we can
make sure there's nothing wrong going on there.

--
Kohsuke Kawaguchi
Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: copying artifacts from slave is slow

Kohsuke Kawaguchi
Administrator
In reply to this post by Rex Chan

Thanks. I see that DeflaterOutputStream is connected directly to the
remoting layer. It's quite possible that it's not sending out data in a
large enough chunk.

Let me do some experiments here.

Rex Chan wrote:

> On Tue, 14 Apr 2009 14:35:00 +1000
> Rex Chan <[hidden email]> wrote:
>
>> Did you mean http://server/hudson/computer/SLAVENAME/systeminfo ?
>
> There's a thread dump section here in systeminfo:
>
> Thread Dump
> Ping thread for channel hudson.remoting.Channel@b82368:channel
>
> java.lang.Thread.sleep(Native Method)
> hudson.remoting.PingThread.run(PingThread.java:64)
>
> Finalizer
>
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> pool-1-thread-2
>
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:474)
> java.lang.UNIXProcess.waitFor(UNIXProcess.java:112)
> hudson.Proc$LocalProc.join(Proc.java:147)
> hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:504)
> hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:485)
> hudson.remoting.UserRequest.perform(UserRequest.java:97)
> hudson.remoting.UserRequest.perform(UserRequest.java:46)
> hudson.remoting.Request$2.run(Request.java:236)
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
> java.util.concurrent.FutureTask.run(FutureTask.java:123)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
> java.lang.Thread.run(Thread.java:613)
>
> main
>
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:474)
> hudson.remoting.Channel.join(Channel.java:568)
> hudson.remoting.Launcher.main(Launcher.java:359)
> hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:308)
> hudson.remoting.Launcher.main(Launcher.java:187)
>
> Channel reader thread: channel
>
> java.io.FileInputStream.readBytes(Native Method)
> java.io.FileInputStream.read(FileInputStream.java:194)
> java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> java.io.BufferedInputStream.read(BufferedInputStream.java:235)
> java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2196)
> java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2486)
> java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2496)
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1273)
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
> hudson.remoting.Channel$ReaderThread.run(Channel.java:665)
>
> Reference Handler
>
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:474)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>
> pool-1-thread-8
>
> java.lang.Thread.dumpThreads(Native Method)
> java.lang.Thread.getAllStackTraces(Thread.java:1452)
> hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:78)
> hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:69)
> hudson.remoting.UserRequest.perform(UserRequest.java:97)
> hudson.remoting.UserRequest.perform(UserRequest.java:46)
> hudson.remoting.Request$2.run(Request.java:236)
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
> java.util.concurrent.FutureTask.run(FutureTask.java:123)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
> java.lang.Thread.run(Thread.java:613)
>
> pool-1-thread-3
>
> java.io.PrintStream.write(PrintStream.java:414)
> java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1685)
> java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1594)
> java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1173)
> java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1127)
> java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1178)
> java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1127)
> java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1178)
> java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1127)
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1284)
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
> hudson.remoting.Channel.send(Channel.java:347)
> hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:112)
> hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:96)
> java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:161)
> java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:118)
> java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:72)
> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
> org.apache.tools.tar.TarBuffer.writeBlock(TarBuffer.java:410)
> org.apache.tools.tar.TarBuffer.writeRecord(TarBuffer.java:388)
> org.apache.tools.tar.TarOutputStream.write(TarOutputStream.java:332)
> hudson.FilePath.writeToTar(FilePath.java:1166)
> hudson.FilePath.access$600(FilePath.java:150)
> hudson.FilePath$28.invoke(FilePath.java:1101)
> hudson.FilePath$28.invoke(FilePath.java:1098)
> hudson.FilePath$FileCallableWrapper.call(FilePath.java:1510)
> hudson.remoting.UserRequest.perform(UserRequest.java:97)
> hudson.remoting.UserRequest.perform(UserRequest.java:46)
> hudson.remoting.Request$2.run(Request.java:236)
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
> java.util.concurrent.FutureTask.run(FutureTask.java:123)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
> java.lang.Thread.run(Thread.java:613)
>
> Signal Dispatcher
>
> ant -file build.xml release-all: stdout copier
>
> java.io.FileInputStream.readBytes(Native Method)
> java.io.FileInputStream.read(FileInputStream.java:194)
> java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
> java.io.BufferedInputStream.read(BufferedInputStream.java:313)
> java.io.FilterInputStream.read(FilterInputStream.java:90)
> hudson.util.StreamCopyThread.run(StreamCopyThread.java:50)
>
> process reaper
>
> java.lang.UNIXProcess.waitForProcessExit(Native Method)
> java.lang.UNIXProcess.access$700(UNIXProcess.java:17)
> java.lang.UNIXProcess$2$1.run(UNIXProcess.java:83)
>

--
Kohsuke Kawaguchi
Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: copying artifacts from slave is slow

Rex Chan
In reply to this post by Kohsuke Kawaguchi

> If the traffic is properly buffered, every packet is almost the full
> size. In normal TCP/IP network that's around 1500 bytes.

I'm not sure if what I'm seeing is useful - I've got it my packet
analyzer sorting by sequence number. I've found on first glance that
packet sizes range from about ~170 bytes for periods to ~1500 bytes as
well.

>
> OTOH, if things are horribly unbuffered, you'll see the TCP payload
> of a few bytes, which should be about 50-60 bytes if you includ
> TCP/IP headers.

--

Rex Chan
-------------------------->  
ish
http://www.ish.com.au
Level 1, 30 Wilson Street Newtown 2042 Australia
phone +61 2 9550 5001   fax +61 2 9550 4001

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: copying artifacts from slave is slow

Ronald Klop
https://hudson.dev.java.net/issues/show_bug.cgi?id=3524

Ngrep shows the packets contain a lot of serialized java meta-data and very little raw data while sending artifacts to the master.
A BufferedOutputStream is used to write larger chunks into the remoting stuff of Hudson. But for some reason the remoting stuff is sending small chunks of data to the server.

Ronald.

Op woensdag, 22 april 2009 09:53 schreef Rex Chan :


> If the traffic is properly buffered, every packet is almost the full
> size. In normal TCP/IP network that's around 1500 bytes.

I'm not sure if what I'm seeing is useful - I've got it my packet
analyzer sorting by sequence number. I've found on first glance that
packet sizes range from about ~170 bytes for periods to ~1500 bytes as
well.

>
> OTOH, if things are horribly unbuffered, you'll see the TCP payload
> of a few bytes, which should be about 50-60 bytes if you includ
> TCP/IP headers.

-- 

Rex Chan
-------------------------->  
ish
http://www.ish.com.au
Level 1, 30 Wilson Street Newtown 2042 Australia
phone +61 2 9550 5001   fax +61 2 9550 4001

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

 


Reply | Threaded
Open this post in threaded view
|

Re: copying artifacts from slave is slow

Kohsuke Kawaguchi
Administrator
Ronald Klop wrote:
> https://hudson.dev.java.net/issues/show_bug.cgi?id=3524
>
> Ngrep shows the packets contain a lot of serialized java meta-data and very little raw data while sending artifacts to the master.
> A BufferedOutputStream is used to write larger chunks into the remoting stuff of Hudson. But for some reason the remoting stuff is sending small chunks of data to the server.

Ah, good point. I should check the Java serialization overhead.

OTOH, again, if things are properly buffered, 8KB of the payload is sent
per one "remoting packet" (with some fixed Java serialization overhead),
so we should be still using most of the bandwidth in actual payload.

> Ronald.
>
> Op woensdag, 22 april 2009 09:53 schreef Rex Chan  :>
>>
>> > If the traffic is properly buffered, every packet is almost the full
>> > size. In normal TCP/IP network that's around 1500 bytes.
>>
>> I'm not sure if what I'm seeing is useful - I've got it my packet
>> analyzer sorting by sequence number. I've found on first glance that
>> packet sizes range from about ~170 bytes for periods to ~1500 bytes as
>> well.
>>
>> >
>> > OTOH, if things are horribly unbuffered, you'll see the TCP payload
>> > of a few bytes, which should be about 50-60 bytes if you includ
>> > TCP/IP headers.
>>
>> --
>>
>> Rex Chan
>> -------------------------->  
>> ish
>> http://www.ish.com.au
>> Level 1, 30 Wilson Street Newtown 2042 Australia
>> phone +61 2 9550 5001   fax +61 2 9550 4001
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>  
>>
>>
>>
>
>

--
Kohsuke Kawaguchi
Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: copying artifacts from slave is slow

Ronald Klop
Hi,

I've experimented some more.
The ProxyOutputStream gives byte[]'s of 512 bytes to the Channel.send() method.

I think the line java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:161) gives 512 byte arrays. If it's possible to put a BufferedOutputStream between the DeflaterOutputStream and the ProxyOutputStream there will be send more bytes at ones I think.

Ronald.

  java.lang.Thread.State: RUNNABLE
  at java.lang.Throwable.getStackTraceElement(Native Method)
  at java.lang.Throwable.getOurStackTrace(Throwable.java:591)
  - locked <0xb159d4c0> (a hudson.remoting.Command$Source)
  at java.lang.Throwable.writeObject(Throwable.java:647)
  - locked <0xb159d4c0> (a hudson.remoting.Command$Source)
  at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:597)
  at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:945)
  at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1461)
  at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
  at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
  at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1509)
  at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1474)
  at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
  at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
  at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326)
  at hudson.remoting.Channel.send(Channel.java:347)
  - locked <0x7957d7a8> (a hudson.remoting.Channel)
  at hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:112)
  - locked <0xaf6fb0b0> (a hudson.remoting.ProxyOutputStream)
  at hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:96)
  at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:161)
  at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:118)
  at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:72)
  - locked <0xaf6fb0f8> (a java.util.zip.GZIPOutputStream)
  at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
  - locked <0xaf6fe7a8> (a java.io.BufferedOutputStream)
  at org.apache.tools.tar.TarBuffer.writeBlock(TarBuffer.java:410)
  at org.apache.tools.tar.TarBuffer.writeRecord(TarBuffer.java:388)
  at org.apache.tools.tar.TarOutputStream.write(TarOutputStream.java:332)
  at hudson.FilePath.writeToTar(FilePath.java:1232)
  at hudson.FilePath.access$600(FilePath.java:152)
  at hudson.FilePath$29.invoke(FilePath.java:1167)
  at hudson.FilePath$29.invoke(FilePath.java:1164)
  at hudson.FilePath$FileCallableWrapper.call(FilePath.java:1583)
  at hudson.remoting.UserRequest.perform(UserRequest.java:97)
  at hudson.remoting.UserRequest.perform(UserRequest.java:46)
  at hudson.remoting.Request$2.run(Request.java:236)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
  at java.util.concurrent.FutureTask.run(FutureTask.java:138)
  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  at hudson.remoting.Engine$1$1.run(Engine.java:54)
  at java.lang.Thread.run(Thread.java:619)



Op vrijdag, 24 april 2009 07:35 schreef Kohsuke Kawaguchi :

Ronald Klop wrote:
> https://hudson.dev.java.net/issues/show_bug.cgi?id=3524
> > Ngrep shows the packets contain a lot of serialized java meta-data and very little raw data while sending artifacts to the master.
> A BufferedOutputStream is used to write larger chunks into the remoting stuff of Hudson. But for some reason the remoting stuff is sending small chunks of data to the server.

Ah, good point. I should check the Java serialization overhead.

OTOH, again, if things are properly buffered, 8KB of the payload is sent per one "remoting packet" (with some fixed Java serialization overhead), so we should be still using most of the bandwidth in actual payload.

> Ronald.
> > Op woensdag, 22 april 2009 09:53 schreef Rex Chan  :> >> >> > If the traffic is properly buffered, every packet is almost the full >> > size. In normal TCP/IP network that's around 1500 bytes.
>> >> I'm not sure if what I'm seeing is useful - I've got it my packet
>> analyzer sorting by sequence number. I've found on first glance that
>> packet sizes range from about ~170 bytes for periods to ~1500 bytes as
>> well. >> >> > >> > OTOH, if things are horribly unbuffered, you'll see the TCP payload
>> > of a few bytes, which should be about 50-60 bytes if you includ
>> > TCP/IP headers.
>> >> -- >> >> Rex Chan
>> -------------------------->  
>> ish
>> http://www.ish.com.au
>> Level 1, 30 Wilson Street Newtown 2042 Australia
>> phone +61 2 9550 5001   fax +61 2 9550 4001
>> >> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>> >>  
>> >> >> > >

-- 
Kohsuke Kawaguchi
Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/
 

 

Reply | Threaded
Open this post in threaded view
|

Re: copying artifacts from slave is slow

Kohsuke Kawaguchi
Administrator

Thanks for investigating this. I put BufferedInput/OutputStream around
this in 1.302.

Ronald Klop wrote:

> Hi,
>
> I've experimented some more.
> The ProxyOutputStream gives byte[]'s of 512 bytes to the Channel.send() method.
>
> I think the line java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:161) gives 512 byte arrays. If it's possible to put a BufferedOutputStream between the DeflaterOutputStream and the ProxyOutputStream there will be send more bytes at ones I think.
>
> Ronald.
>
>   java.lang.Thread.State: RUNNABLE
>   at java.lang.Throwable.getStackTraceElement(Native Method)
>   at java.lang.Throwable.getOurStackTrace(Throwable.java:591)
>   - locked <0xb159d4c0> (a hudson.remoting.Command$Source)
>   at java.lang.Throwable.writeObject(Throwable.java:647)
>   - locked <0xb159d4c0> (a hudson.remoting.Command$Source)
>   at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:945)
>   at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1461)
>   at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
>   at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
>   at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1509)
>   at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1474)
>   at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
>   at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
>   at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326)
>   at hudson.remoting.Channel.send(Channel.java:347)
>   - locked <0x7957d7a8> (a hudson.remoting.Channel)
>   at hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:112)
>   - locked <0xaf6fb0b0> (a hudson.remoting.ProxyOutputStream)
>   at hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:96)
>   at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:161)
>   at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:118)
>   at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:72)
>   - locked <0xaf6fb0f8> (a java.util.zip.GZIPOutputStream)
>   at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
>   - locked <0xaf6fe7a8> (a java.io.BufferedOutputStream)
>   at org.apache.tools.tar.TarBuffer.writeBlock(TarBuffer.java:410)
>   at org.apache.tools.tar.TarBuffer.writeRecord(TarBuffer.java:388)
>   at org.apache.tools.tar.TarOutputStream.write(TarOutputStream.java:332)
>   at hudson.FilePath.writeToTar(FilePath.java:1232)
>   at hudson.FilePath.access$600(FilePath.java:152)
>   at hudson.FilePath$29.invoke(FilePath.java:1167)
>   at hudson.FilePath$29.invoke(FilePath.java:1164)
>   at hudson.FilePath$FileCallableWrapper.call(FilePath.java:1583)
>   at hudson.remoting.UserRequest.perform(UserRequest.java:97)
>   at hudson.remoting.UserRequest.perform(UserRequest.java:46)
>   at hudson.remoting.Request$2.run(Request.java:236)
>   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>   at hudson.remoting.Engine$1$1.run(Engine.java:54)
>   at java.lang.Thread.run(Thread.java:619)
>
>
>
>
> Op vrijdag, 24 april 2009 07:35 schreef Kohsuke Kawaguchi  :>
>> Ronald Klop wrote:
>> > https://hudson.dev.java.net/issues/show_bug.cgi?id=3524
>> > > Ngrep shows the packets contain a lot of serialized java meta-data and very little raw data while sending artifacts to the master.
>> > A BufferedOutputStream is used to write larger chunks into the remoting stuff of Hudson. But for some reason the remoting stuff is sending small chunks of data to the server.
>>
>> Ah, good point. I should check the Java serialization overhead.
>>
>> OTOH, again, if things are properly buffered, 8KB of the payload is sent per one "remoting packet" (with some fixed Java serialization overhead), so we should be still using most of the bandwidth in actual payload.
>>
>> > Ronald.
>> > > Op woensdag, 22 april 2009 09:53 schreef Rex Chan  :> >> >> > If the traffic is properly buffered, every packet is almost the full >> > size. In normal TCP/IP network that's around 1500 bytes.
>> >> >> I'm not sure if what I'm seeing is useful - I've got it my packet
>> >> analyzer sorting by sequence number. I've found on first glance that
>> >> packet sizes range from about ~170 bytes for periods to ~1500 bytes as
>> >> well. >> >> > >> > OTOH, if things are horribly unbuffered, you'll see the TCP payload
>> >> > of a few bytes, which should be about 50-60 bytes if you includ
>> >> > TCP/IP headers.
>> >> >> -- >> >> Rex Chan
>> >> -------------------------->  
>> >> ish
>> >> http://www.ish.com.au
>> >> Level 1, 30 Wilson Street Newtown 2042 Australia
>> >> phone +61 2 9550 5001   fax +61 2 9550 4001
>> >> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: [hidden email]
>> >> For additional commands, e-mail: [hidden email]
>> >> >>  
>> >> >> >> > >
>>
>> --
>> Kohsuke Kawaguchi
>> Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/
>>  
>>
>>
>>
>>  
>
>

--
Kohsuke Kawaguchi
Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/

smime.p7s (4K) Download Attachment