Comment 1 Scott Lewis 2011-06-17 21:23:35 EDT

Hmmm.  This appears to be something wrong with the JRE implementation of ftp.  

The reason I say this is two-fold:

1) No changes in any ECF filetransfer have take place in this code in the past year (at least...perhaps longer)
2) The stack trace shows the exception is coming from

...
Caused by: java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
    at
sun.net.TransferProtocolClient.readServerResponse(TransferProtocolClient.java:49)
    at sun.net.ftp.FtpClient.readReply(FtpClient.java:217)
    at sun.net.ftp.FtpClient.issueCommand(FtpClient.java:193)
    at sun.net.ftp.FtpClient.login(FtpClient.java:509)
    at
sun.net.www.protocol.ftp.FtpURLConnection.connect(FtpURLConnection.java:276)
    at
sun.net.www.protocol.ftp.FtpURLConnection.getInputStream(FtpURLConnection.java:352)
    at

One question that may provide some useful information:  what version of the JRE was used for this test?  Is it possible that the JRE implementers changed something about the login process (for anonymous login?) in this version (as it appears to be something about the login response reading).

I will also attempt to reproduce.  Recent...before 3.5.1...test runs haven't shown this error, however...and we do run a few ftp tests in our continuous build/test cycle.

Comment 2 Scott Lewis 2011-06-17 22:10:36 EDT

I'm currently able to run the ECF ftp test just fine (with latest ECF/3.5.1 and JRE 1.6).

The test case that we use downloads a file via ftp using the following URL:

ftp://ftp.osuosl.org/pub/eclipse/rt/ecf/3.2/3.6/site.p2/features/org.eclipse.ecf.core_3.2.0.v20100219-1253.jar

Obviously this is reading a file rather than a directory.

Comment 3 David Williams 2011-06-19 16:04:16 EDT

I originally asked about this issue on p2-dev list, to figure out what to use, and what to advise others to use, in the p2.mirrorsURL property. So, just to cross reference, the following page should be updated once this is sorted out. 

http://wiki.eclipse.org/Equinox/p2/p2.mirrorsURL#What_to_add.3F

Comment 4 Scott Lewis 2011-06-20 15:21:49 EDT

I now suspect that this is a simple matter of the JRE FTP impl not supporting the (anonymous) browsing of directories...so this URI:

ftp://ftp.osuosl.org/pub/eclipse/eclipse/updates/3.6/

doesn't work because the directory browse fails...i.e. org.eclipse.ecf.provider.filetransfer.browse.URLFileSystemBrowser.runRequest(URLFileSystemBrowser.java:115).

Does p2 have some way to specify the direct *file* for repository access?...e.g.:

ftp://ftp.osuosl.org/pub/eclipse/eclipse/updates/3.6/content.xml

or

ftp://ftp.osuosl.org/pub/eclipse/eclipse/updates/3.6/content.jar

or something else?  I think that in previous versions of p2 it was possible to do this, but I' m not sure if it still is in the new p2 UI.  If so, I believe that the FTP access would/will work by specifying the entire file (as long as the directory browse access via ftp is skipped).

If FTP file browse is needed it could be very easily introduced (e.g. through new/non-jre-based provider), but such an enhancement would require some resources allocated to transport enhancements.

Comment 5 David Williams 2012-02-24 12:51:52 EST

So ... what's the status of "ftp support" for p2? 

Should this bug be moved "back" to p2? 

This issue same up again in bug 372504 ... that is, does p2.mirrorsURL need to be limited to "http" or will both http and ftp work ok? Our instructions at 
http://wiki.eclipse.org/Equinox/p2/p2.mirrorsURL#What_to_add.3F
still say to use protocol=http, but ... they also say to watch this bug for "progress" (and, there has been no progress, for 9 months ... so .... just asking what's next?)

Comment 6 Scott Lewis 2012-02-24 13:07:41 EST

(In reply to comment #5)
> So ... what's the status of "ftp support" for p2? 
> 
> Should this bug be moved "back" to p2? 
> 
> This issue same up again in bug 372504 ... that is, does p2.mirrorsURL need to
> be limited to "http" or will both http and ftp work ok? Our instructions at 
> http://wiki.eclipse.org/Equinox/p2/p2.mirrorsURL#What_to_add.3F
> still say to use protocol=http, but ... they also say to watch this bug for
> "progress" (and, there has been no progress, for 9 months ... so .... just
> asking what's next?)

Hi David.

The summary is that p2 currently needs/uses the ability to browse a directory structure.  The ECF ftp provider (based upon JRE) does not support file browse (rather it only supports file download).

So two possibilities that I can think of:  

1) p2 is modified to not browse (with ftp)...and then the current provider will work (file download only)

2) A new ECF ftp provider is created that supports browse (probably by using some non JRE-based ftp impl...e.g. apache.net ftp)

Unfortunately, ECF currently doesn't have any resources to commit to 2...which is why we haven't done it.  It is not a huge job technically...given that there are ftp impls in orbit, and the ECF provider code is simple.

Comment 7 Ian Bull 2012-02-24 13:34:49 EST

(In reply to comment #6)
> Hi David.
> 
> The summary is that p2 currently needs/uses the ability to browse a directory
> structure.  The ECF ftp provider (based upon JRE) does not support file browse
> (rather it only supports file download).
> 

Scott, do you know why p2 needs the needs/uses the 'browse'?  (I realize that's probably a question I should know the answer to :-) ).  I thought we (p2) just tries different URIs and work on the ones that return content.  Are things different when FTP is used?  

Transport is not my area of expertise.  

I've CC'd Henrik here. He seems to be very knowledgeable when it comes to p2 and transport.

Comment 8 Scott Lewis 2012-02-24 13:39:52 EST

(In reply to comment #7)
> (In reply to comment #6)
> > Hi David.
> > 
> > The summary is that p2 currently needs/uses the ability to browse a directory
> > structure.  The ECF ftp provider (based upon JRE) does not support file browse
> > (rather it only supports file download).
> > 
> 
> Scott, do you know why p2 needs the needs/uses the 'browse'?  (I realize that's
> probably a question I should know the answer to :-) ).  I thought we (p2) just
> tries different URIs and work on the ones that return content.  Are things
> different when FTP is used?  

I don't know why p2 uses the browse.  I sort of suspect that it's because it's reading directory contents of repo...in order to read one of content.jar, content.xml, etc....but I'm not certain.  I doubt things are different when p2 is used...but again I'm not certain.

> 
> Transport is not my area of expertise.  
> 
> I've CC'd Henrik here. He seems to be very knowledgeable when it comes to p2
> and transport.

Henrik did do a lot of the work on the p2 transport.

Comment 9 Thomas Hallgren 2012-02-24 16:45:11 EST

(In reply to comment #8)
> I don't know why p2 uses the browse.  I sort of suspect that it's because it's
> reading directory contents of repo...in order to read one of content.jar,
> content.xml, etc....but I'm not certain.  I doubt things are different when p2
> is used...but again I'm not certain.
> 
Something is very wrong if p2 uses the browse capability. It knows exactly what files to look for and will specify them in its request. AFAIK, it never browse anything. In fact, most repositories accessed over http does not allow browsing so if it did, that would be a major problem, not just with ftp.

Comment 10 Scott Lewis 2012-02-24 17:11:07 EST

(In reply to comment #9)
> (In reply to comment #8)
> > I don't know why p2 uses the browse.  I sort of suspect that it's because it's
> > reading directory contents of repo...in order to read one of content.jar,
> > content.xml, etc....but I'm not certain.  I doubt things are different when p2
> > is used...but again I'm not certain.
> > 
> Something is very wrong if p2 uses the browse capability. It knows exactly what
> files to look for and will specify them in its request. AFAIK, it never browse
> anything. In fact, most repositories accessed over http does not allow browsing
> so if it did, that would be a major problem, not just with ftp.

Hi Thomas.

I was previously just looking at the original stack track from Pascal...which has the ECF browse access in the stack:

...
sun.net.www.protocol.ftp.FtpURLConnection.connect(FtpURLConnection.java:276)
    at
sun.net.www.protocol.ftp.FtpURLConnection.getInputStream(FtpURLConnection.java:352)
    at
org.eclipse.ecf.provider.filetransfer.browse.URLFileSystemBrowser.runRequest(URLFileSystemBrowser.java:115)
    at
org.eclipse.ecf.provider.filetransfer.browse.AbstractFileSystemBrowser$DirectoryJob.run(AbstractFileSystemBrowser.java:69


But perhaps this is no longer being used by p2...as when I put the following update site into p2

ftp://ftp.osuosl.org/pub/eclipse/eclipse/updates/3.6/R-3.6-201006080911/

with p2 3.7.1...it reads the repo just fine (!)

OTOH, if I go to

ftp://ftp.osuosl.org/pub/eclipse/eclipse/updates/3.6

with 3.7.1 it does not throw an exception (as pascal indicated)...but rather shows nothing in the repo.  When I go to ...eclipse/updates/3.6 with web browser it shows that it has compositeArtifacts.jar and compositeContent.jar, so perhaps that's why it shows nothing.

So I guess at this point (3.7.1) I can't reproduce the problem originally reported by Pascal.  Are others getting other/different behavior?

Comment 11 David Williams 2012-02-24 22:16:17 EST

Well, I am glad I asked.

I did some tests, and I'm willing to resolve this as "worksforme", and fear ftp no more. Not sure if something in p2 changed ... or the original exceptions were due to some other subtle issues, but I did some tests updating Indigo SR1 to Indigo SR2 using FTP sites and encountered no issues. 

I was able to do a realistic test because the .../releases/maintenance we used for staging SR1 is mirrored, even though it really should not be (bug 372156) so I used that to "experiment" with the protocol attribute, setting it to ftp (to use only ftp sites for SR2 aggregated updates) and omitting it completely (so both ftp and http would be fetched). In both cases I was able to update Indigo SR1 Java EE IDE package to SR2, using only following repo sites (and disabling the rest). 

http://download.eclipse.org/releases/maintenance/
http://download.eclipse.org/technology/epp/packages/indigo/

Since we "point to" the platform's maintenance artifacts via composite, there would still be many "http" fetches expected (since dropping back to 'download.eclipse.org'), even in the "ftp only" case, but there was enough ftp sites hit for other project's artifacts that I am confident there is not a general problem with using ftp sites, at least if starting with Indigo SR1 ... and, with debug/trace turned on, to capture where downloads came from. :) 

The "hits" to mirrors for Java EE IDE update was as follows: 

$ ~/builds/scripts/analyzep2mirrors.sh 
    423 http://download.eclipse.org
    233 ftp://mirror.cc.vt.edu
    116 ftp://mirror.cc.columbia.edu
     69 ftp://eclipse.mirrors.tds.net
     51 ftp://ftp.gtlib.gatech.edu
     31 ftp://ftp.osuosl.org
     15 ftp://ftp.ussg.iu.edu
      3 ftp://mirror.selfnet.de
      1 ftp://carroll.aset.psu.edu

So, approximately 400 fetches from 8 ftp sites ... no exceptions in logs. 

I'll update the related wiki and bug bug 372504 with this new information. 

Naturally, if any of the p2/ecf experts want to investigate deeper ... be my guest, but seems to me to work just fine in general, and recommend we wait for it to happen again to figure out what special circumstances might give rise to the original issue. 

Thanks to everyone involved, your quick responses motivated me to test it empirically.

Comment 12 David Williams 2012-02-24 22:42:44 EST

I should also document, I tested with IBM's JRE 1.6 and 1.7 ... I suppose there could be some sort of VM difference (or bug?) but, if anyone is interested in testing, with other JREs or for other reasons, I have left the .../releases/maintenance repo with protocol=ftp for now. I would like to remove that staging repo soon (in week or so) so if anyone wants it to stay around longer, for testing, be sure to say so. 

For what its worth, in my final test, using Java 6, still ftp only, I did get a very different "number" of updates ... not sure if this is a problem with my procedure (multiple logs?) or my "histogram script" ... but, thought I'd note it, in case anyone does try to duplicate the test and sees different results. My latest test showed: 

    145 ftp://ftp.osuosl.org
    127 http://download.eclipse.org
     54 ftp://mirror.cc.columbia.edu
     51 ftp://eclipse.mirrors.tds.net
     27 ftp://ftp.cse.buffalo.edu
      9 ftp://ftp.ussg.iu.edu
      1 ftp://ftp.gtlib.gatech.edu

The script I used to "scrape" the log was inspired 
by bug 372504 comment 9, I ended up with script below, where "fulllog.txt" is where I captured the output from having 
.options file of 
 org.eclipse.equinox.p2.core/debug=true
 org.eclipse.equinox.p2.core/artifacts/mirrors=true


grep Selected fulllog.txt | cut -d ":" -f 5- | cut -d "(" -f 2 | cut -d "/" -f 1,2,3 |  sort | uniq -c | sort -r -n 

(so could be bugs in that?) 

Thanks again,

Comment 13 David Williams 2012-02-25 13:27:54 EST

One final note for those interested in side-issue details: I did update the artifact.jar/xml file for Indigo SR2, removed the "protocol" so ftp sites would be returned too, and repeated the test upgrading from SR1 Java EE IDE to SR2 version (using all the built-in sites). 

It appears "ftp" is not used very often. Only 24 times, from 
ftp://eclipse.c3sl.ufpr.br
Not sure if that is because http is inherently faster than ftp? So p2 is doing a good job of figuring out best sites to use? Or, maybe a quirk of my particular location (North Carolina). 

     92 http://mirror.cc.columbia.edu
     54 http://mirrors.med.harvard.edu
     50 http://mirror.cc.vt.edu
     43 http://mirrors.ibiblio.org
     43 http://carroll.aset.psu.edu
     26 http://download.eclipse.org
     24 ftp://eclipse.c3sl.ufpr.br
     20 http://ftp.jaist.ac.jp
     17 http://mirrors.xmission.com
     16 http://ftp.ussg.iu.edu
     13 http://www.gtlib.gatech.edu
     11 http://ftp.osuosl.org
      8 http://mirror.aarnet.edu.au
      8 http://ftp.yz.yamagata-u.ac.jp
      5 http://eclipse.mirrorcatalogs.com
      3 http://mirrors.ustc.edu.cn
      1 http://eclipse.stu.edu.tw


But, thanks again for all comments and help figuring this out.

Comment 14 Scott Lewis 2012-02-25 15:43:12 EST

(In reply to comment #13)

Thanks David for all the testing...and others for help with this.  I suspect that the original problem occurred because of some specific combination/version of p2, jre, and ftp mirror sites...but in any case I'm glad that things are working now.