Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 546496

Summary: Problems cloning the git repository eclipse.platform.common
Product: Community Reporter: Luis Colorado <luiscoloradourcola>
Component: GitAssignee: Eclipse Webmaster <webmaster>
Status: RESOLVED WORKSFORME QA Contact:
Severity: normal    
Priority: P3 CC: denis.roy
Version: unspecified   
Target Milestone: ---   
Hardware: PC   
OS: other   
Whiteboard:

Description Luis Colorado CLA 2019-04-17 00:39:22 EDT
I have just started to clone the eclipse.platform.releng.aggregator repository in order to try to port eclipse natively to FreeBSD, but I'm having problems cloning some of the submodules from that project.  Basically, after a while of full speed downloading, the connection throttles and finally it ends up aborting the download.  I think this happens mainly on the largest (or busiest) submodules.  After some retrying, I have had some success with some, but some others resist.  Can anybody explain me if this behaviour is normal, do I have to change something?  Is there some limit on the amount of downloading possible from the git repositories?  Is there any alternative to clone from?

Thanks in advance,

Luis
Comment 1 Denis Roy CLA 2019-04-17 09:25:27 EDT
Can you provide insight as to which submodules, or an output of your git clone command?
Comment 2 Luis Colorado CLA 2019-04-17 23:31:41 EDT
The command was:
git pull --recurse-submodules

Other repositories had failures also, but after repeating sevaral time I got these only failing...

(sorry for the locale in spanish :( )
------------------------------------------------->8---------------------------------------------
[...]
Clonando en '/home/lcu/eclipse_build/eclipse.platform.releng.aggregator/eclipse.platform.common'...
Fssh_packet_write_wait: Connection to 198.41.30.196 port 29418: Broken pipe
fatal: el remoto se colgó de manera inesperada 
fatal: EOF temprano
fatal: index-pack falló
fatal: clon de 'ssh://lcoloradow56@git.eclipse.org:29418/platform/eclipse.platform.common' en la ruta de submódulo '/home/lcu/eclipse_build/eclipse.platform.releng.aggregator/eclipse.platform.common' falló
Falló al clonar 'eclipse.platform.common'. Reintento programado
Clonando en '/home/lcu/eclipse_build/eclipse.platform.releng.aggregator/eclipse.platform.swt.binaries'...
Fssh_packet_write_wait: Connection to 198.41.30.196 port 29418: Broken pipe
fatal: el remoto se colgó de manera inesperada
fatal: EOF temprano
fatal: index-pack falló
fatal: clon de 'ssh://lcoloradow56@git.eclipse.org:29418/platform/eclipse.platform.swt.binaries.git' en la ruta de submódulo '/home/lcu/eclipse_build/eclipse.platform.releng.aggregator/eclipse.platform.swt.binaries' falló
Falló al clonar 'eclipse.platform.swt.binaries'. Reintento programado
Clonando en '/home/lcu/eclipse_build/eclipse.platform.releng.aggregator/eclipse.platform.ui'...
Fssh_packet_write_wait: Connection to 198.41.30.196 port 29418: Broken pipe
fatal: el remoto se colgó de manera inesperada
fatal: EOF temprano
fatal: index-pack falló
fatal: clon de 'ssh://lcoloradow56@git.eclipse.org:29418/platform/eclipse.platform.ui' en la ruta de submódulo '/home/lcu/eclipse_build/eclipse.platform.releng.aggregator/eclipse.platform.ui' falló
Falló al clonar 'eclipse.platform.ui'. Reintento programado
Clonando en '/home/lcu/eclipse_build/eclipse.platform.releng.aggregator/eclipse.platform.common'...
Fssh_packet_write_wait: Connection to 198.41.30.196 port 29418: Broken pipe
fatal: el remoto se colgó de manera inesperada
fatal: EOF temprano
fatal: index-pack falló
fatal: clon de 'ssh://lcoloradow56@git.eclipse.org:29418/platform/eclipse.platform.common' en la ruta de submódulo '/home/lcu/eclipse_build/eclipse.platform.releng.aggregator/eclipse.platform.common' falló
Falló al clonar 'eclipse.platform.common' una segunda vez, abortando
------------------------------------------------->8---------------------------------------------
Comment 3 Luis Colorado CLA 2019-04-18 00:22:19 EDT
trying to clone individually those repos, the bandwidth allowed begins decreasing at over 10Mb transferred and lowers and lowers until it stalls at about 70% of the total download:

------------------------------------------------->8--------------------------------------------
$ LANG=C git clone -v ssh://lcoloradow56@git.eclipse.org:29418/platform/eclipse.platform.common
Cloning into 'eclipse.platform.common'...
remote: Counting objects: 4314, done
remote: Finding sources: 100% (71/71)
Receiving objects:  70% (153215/217531), 92.47 MiB | 349.00 KiB/s   
^Cfatal: the remote end hung up unexpectedly

------------------------------------------------->8--------------------------------------------
(this time I interrupted it, but it had stalled at that point)
Comment 4 Luis Colorado CLA 2019-04-18 01:17:33 EDT
by cloning from github and then changing the origin in the different submodules, I have finally succeeded in the downloading of all the submodules.

Anyway, there should be a traffic profiler that limits the bandwidth to 300Kbps, but after a while it forgets the state and cuts the traffic to 0.  No problem on fetching an already downloaded repo.  But for the first time clone, the repository at github has been more helpful.
Comment 5 Luis Colorado CLA 2019-04-18 01:18:14 EDT
Anyway, you can close the ticket if you wish.
Comment 6 Denis Roy CLA 2019-04-18 09:23:57 EDT
> Anyway, there should be a traffic profiler that limits the bandwidth to
> 300Kbps, but after a while it forgets the state and cuts the traffic to 0. 
> No problem on fetching an already downloaded repo.  But for the first time
> clone, the repository at github has been more helpful.

We have no such mechanism to profile bandwidth.

I cloned from an Azure host in Germany, and there is a noticeable slowdown at 67%. There is likely a large binary object that git struggles to resolve or compress.  After it passes 70% it picks up to regular speed until completion.

I ran a garbage collection on the repo and it didn't do much.