Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 361395

Summary: Really slow network connection
Product: Community Reporter: Chris Frost <eclipse>
Component: ServersAssignee: Eclipse Webmaster <webmaster>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: P3 CC: hsiliev
Version: unspecified   
Target Milestone: ---   
Hardware: PC   
OS: Mac OS X - Carbon (unsup.)   
Whiteboard:

Description Chris Frost CLA 2011-10-19 09:21:35 EDT
I'm trying to run a Virgo build locally and it can normally upload the artifacts as it goes pretty quick but its going slower and slower and slower today and I can't get the build through.
Comment 1 Denis Roy CLA 2011-10-19 09:28:39 EDT
Can you help me understand what "pretty quick" and "slower" mean, in terms of numbers?  Also, to what host are you uploading?

Also, I'd need your remote IP address to see if there's anything going on across the world.  Feel free to email it to webmaster@eclipse.org
Comment 2 Denis Roy CLA 2011-10-19 10:04:41 EDT
Looks like we're picking up some major lag on the backbone between Toronto and the UK:

 8  bb1.tor.primus.ca (195.69.145.154)  11.602 ms   11.223 ms   10.470 ms
 9  ams-ix-1.enta.net (195.69.145.82)  177.651 ms   177.412 ms   178.869 ms
Comment 3 Chris Frost CLA 2011-10-19 10:18:16 EDT
How did you detect this, a colleague in Bulgaria is having the problem. Guess it could be the same pipes causing it.
Comment 4 Denis Roy CLA 2011-10-19 10:34:17 EDT
A tcp traceroute from build.eclipse.org to your location.  If you send me their IP in Bulgaria I can have a look.  However, it's pretty far down the route, and there's probably not much I, or our ISP can do about it.
Comment 5 Hristo Iliev CLA 2011-10-19 12:51:03 EDT
Hi,

I get a log of timeouts trying to clone git repos.

My IP is 91.92.153.114.

From my side:

PING git.eclipse.org (206.191.52.44) 56(84) bytes of data.
64 bytes from 206.191.52.44: icmp_req=1 ttl=111 time=161 ms
64 bytes from 206.191.52.44: icmp_req=2 ttl=111 time=161 ms
64 bytes from 206.191.52.44: icmp_req=3 ttl=111 time=161 ms

traceroute to git.eclipse.org (206.191.52.44), 30 hops max, 60 byte packets
 1  router (192.168.1.1)  0.742 ms  1.001 ms  1.335 ms
 2  91.92.153.113 (91.92.153.113)  4.921 ms  4.914 ms  5.089 ms
 3  rtr-int.cores.bg (85.187.217.193)  6.923 ms  7.097 ms  7.089 ms
 4  95-43-228-233.btc-net.bg (95.43.228.233)  7.452 ms  7.690 ms  7.686 ms
 5  193.231.106.37 (193.231.106.37)  12.205 ms  12.379 ms  12.981 ms
 6  10.0.241.133 (10.0.241.133)  58.040 ms  54.701 ms  54.935 ms
 7  10.0.244.165 (10.0.244.165)  56.587 ms  56.583 ms  56.627 ms
 8  10.0.200.14 (10.0.200.14)  58.340 ms  58.336 ms  58.562 ms
 9  10.0.200.38 (10.0.200.38)  56.787 ms  54.438 ms  56.162 ms
10  rc2ec.il.shawcable.net (195.69.144.148)  156.567 ms  158.322 ms  158.116 ms
11  66.163.65.1 (66.163.65.1)  168.157 ms  162.658 ms  163.622 ms
12  rc2sh-pos3-0.mt.shawcable.net (66.163.76.146)  160.273 ms  159.245 ms  159.822 ms
13  ra1fs-tge2-3.mt.shawcable.net (66.163.66.122)  160.301 ms  160.788 ms  161.005 ms
14  rx0sh-mountain-cable.mt.bigpipeinc.com (66.244.223.238)  155.685 ms  155.675 ms  155.817 ms
15  10.201.120.133 (10.201.120.133)  165.084 ms  165.052 ms  162.972 ms
16  10.201.120.77 (10.201.120.77)  165.011 ms  164.709 ms  164.436 ms
17  10.201.122.194 (10.201.122.194)  165.405 ms  163.798 ms  168.639 ms
18  206.191.0.111 (206.191.0.111)  164.608 ms  164.812 ms  165.133 ms

Regards,
Hristo Iliev
Comment 6 Denis Roy CLA 2011-10-19 15:28:07 EDT
All is well up to Romania:
>  5  193.231.106.37 (193.231.106.37)  12.205 ms  12.379 ms  12.981 ms

This next hop somewhere between Romania and Amsterdam is adding 40+ ms.  I can't trace it since it's a private IP.
>  6  10.0.241.133 (10.0.241.133)  58.040 ms  54.701 ms  54.935 ms
>  7  10.0.244.165 (10.0.244.165)  56.587 ms  56.583 ms  56.627 ms
>  8  10.0.200.14 (10.0.200.14)  58.340 ms  58.336 ms  58.562 ms
>  9  10.0.200.38 (10.0.200.38)  56.787 ms  54.438 ms  56.162 ms

Amsterdam -- ouch.  +100ms
> 10  rc2ec.il.shawcable.net (195.69.144.148)  156.567 ms  158.322 ms  158.116 ms

The hop from Amsterdam to Canada is only adding 10ms, which is nice to see.
> 11  66.163.65.1 (66.163.65.1)  168.157 ms  162.658 ms  163.622 ms

The rest of the pipes in Canada aren't adding any significant lag.
> 18  206.191.0.111 (206.191.0.111)  164.608 ms  164.812 ms  165.133 ms


From what I can tell, some of the European/atlantic pipes are pretty busy.
Comment 7 Denis Roy CLA 2011-10-25 11:58:02 EDT
Things are much worse today:

 9  bb1.tor.primus.ca (195.69.145.154)  18.772 ms   17.752 ms   18.275 ms
10  ams-ix-1.enta.net (195.69.145.82)  184.808 ms   186.518 ms   186.507 ms


I've opened a ticket with our ISP (primus.ca) to see if they can change the routing.  That lag is ridiculous.
Comment 8 Denis Roy CLA 2011-10-25 12:05:57 EDT
> Amsterdam -- ouch.  +100ms
> > 10  rc2ec.il.shawcable.net (195.69.144.148)  156.567 ms  158.322 ms  158.116 ms

So the common link between Chris' lag and Hristo's lag is that it happens in Amsterdam -- albeit from two different routes.

I've forwarded this finding to our ISP.  They should be able to route differently from Toronto.
Comment 9 Chris Frost CLA 2011-10-25 12:07:13 EDT
Hi,

Thank you for keeping an eye on this. It is appreciated.

Chris.
Comment 10 Denis Roy CLA 2011-10-25 16:54:29 EDT
(In reply to comment #8)
> > Amsterdam -- ouch.  +100ms
> > > 10  rc2ec.il.shawcable.net (195.69.144.148)  156.567 ms  158.322 ms  158.116 ms


Our ISP confirms a new gateway has been selected.  Looks like hop #10 is now a lot faster.
10  ams-ix-1.enta.net (195.69.145.82)  98.411 ms   97.991 ms   99.117 ms

This may not be earth shattering, but the extra 60ms per packet was contributing to lots of lag.

I'll keep an eye on it for a few days to make sure it's not just a time-of-day thing.  If you can report back your observations that would be good.
Comment 11 Chris Frost CLA 2011-10-26 05:26:05 EDT
Hi,

At the same time as you sent comment 8 I had a local build running and it sped through with no problems. It makes me think the network lag is a different issue unless the ISP had already switched the route.

Either way, things are looking much better now, thank you.

Chris.
Comment 12 Hristo Iliev CLA 2011-10-26 07:09:57 EDT
I was able to clone all Virgo repos without problems. Looks better now.


traceroute to git.eclipse.org (206.191.52.44), 30 hops max, 60 byte packets
 1  router (192.168.1.1)  0.896 ms  1.128 ms  1.382 ms
 2  91.92.153.113 (91.92.153.113)  5.374 ms  5.367 ms  5.558 ms
 3  rtr-int.cores.bg (85.187.217.193)  6.811 ms  6.982 ms  6.975 ms
 4  95-43-228-233.btc-net.bg (95.43.228.233)  7.591 ms  7.611 ms  7.572 ms
 5  193.231.106.37 (193.231.106.37)  14.524 ms  14.324 ms  14.503 ms
 6  10.0.241.133 (10.0.241.133)  57.936 ms  53.684 ms  54.097 ms
 7  10.0.244.165 (10.0.244.165)  56.121 ms  56.113 ms  56.349 ms
 8  10.0.200.14 (10.0.200.14)  58.331 ms  58.598 ms  58.590 ms
 9  10.0.200.38 (10.0.200.38)  54.451 ms  52.851 ms  53.959 ms
10  bb1.tor.primus.ca (195.69.145.154)  142.912 ms  142.907 ms  143.137 ms
11  10.201.120.129 (10.201.120.129)  152.201 ms  151.491 ms  154.223 ms
12  10.201.120.77 (10.201.120.77)  152.547 ms  154.434 ms  156.885 ms
13  10.201.122.194 (10.201.122.194)  153.002 ms  154.396 ms  157.650 ms
14  206.191.0.111 (206.191.0.111)  153.659 ms  153.186 ms  150.437 ms
Comment 13 Denis Roy CLA 2012-03-23 11:21:59 EDT
Chris, Hristo, are you still affected by this, or have the routing and peering changes helped?
Comment 14 Chris Frost CLA 2012-03-23 11:27:14 EDT
Hi,

It's been OK for me recently so I guess so.
Comment 15 Denis Roy CLA 2012-03-23 11:48:53 EDT
Ok, let's mark this fixed.