Bug 319907

Comment 1 Chris Austin 2010-11-30 10:37:45 EST

We have 2 of the servlets done and are currently working on a validation schema.

Comment 2 Chris Goldthorpe 2011-01-11 16:32:34 EST

Can we get a status update on this plan item - do you have a current version of the API that you can attach? Do you have an estimate of when a full patch would be available for review?

Comment 3 Snehasish Paul 2011-01-12 13:28:47 EST

I will attach the current version patch by coming Monday (01/17/11).
Also I will try tp provide the estimate of when full patch would be available on Monday.

Comment 4 Snehasish Paul 2011-01-17 10:49:08 EST

Created attachment 186915 [details]
Draft of Public API services

Patch contains changes at org.eclipse.help.webapp and org.eclipse.ua.tests plugins. There are 2 testcases failing current. I will update the fix for the same shortly.

Comment 5 Chris Goldthorpe 2011-01-17 19:41:55 EST

Is there a document which describes the servlet API created by this patch? This would include a description of how the API might be used and what parameters are valid. I saw that there was a new extension point org.eclipse.help.webapp.secureServlet but I could not tell how it worked.

Another point to note is that as you work for IBM your contribution is covered under the copyright text:

 * Copyright (c) 2011 IBM Corporation and others.
 * All rights reserved. This program and the accompanying materials
 * are made available under the terms of the Eclipse Public License v1.0
 * which accompanies this distribution, and is available at
 * http://www.eclipse.org/legal/epl-v10.html
 * 
 * Contributors:
 *     IBM Corporation - initial API and implementation

so there is no need to add your name as a contributor, we only do this when the contributor does not work for the corporation listed at the top of the copyright.

Comment 6 Rupesh Kumar 2011-01-18 04:51:10 EST

We are facing small issue with xml data generated from the TocServlet. <toc> node contains three types of child nodes <anchor>, <topic> and <enablement>. Problem I am facing is when we get the random ordered element data for <anchor> and <topic>. W3C XML schema does not allow random data declaration. 

Example:
<anchor id="c_afterworkbench" />
<topic href="concepts/concepts-4.htm" label="Perspectives" />
<anchor id="c_afterperspectives" /> 
<topic href="concepts/concepts-6.htm" label="Editors"/>
<anchor id="c_aftereditors" />
<topic href="concepts/concepts-5.htm" label="Views"/>

Please provide me your suggestions.

Comment 7 Rupesh Kumar 2011-01-19 12:41:51 EST

Few more points about the xml data generated by tocServlet. I saw some more example data where tocServlet generate very unexpected xml elements attributes names like unknownAttribute, unknownAttribute2 and aegraerg. I also found very unusual xml data structure.
Example:
<test>
   <topic label="Tasks">
   <aergaer />
	<test2 attr="true">
	  <gaergr />
	  <link toc="data/help/toc/extraContent/simple_toc.xml" />
	</test2>
  </topic>
</test>

I believe the xml data generated by the tocServlet is very random. In my opinion we should not go for test xml schema for tocServlet. 

Please let us know your thoughts.

Comment 8 Chris Goldthorpe 2011-01-19 13:36:59 EST

The TocServlet returns the contents of the Toc files and these are not filtered to remove unused elements or attributes. These entries are coming directly from /org.eclipse.ua.tests/data/help/toc/extraContent/toc.xml.

Comment 9 Chris Austin 2011-01-19 15:05:37 EST

Created attachment 187144 [details]
Patch with Updated Validation Servlet documentation

Comment 10 Snehasish Paul 2011-01-21 09:53:34 EST

Created attachment 187289 [details]
Public API services

Fixed all the known errors, removed all reference to "/help" context root.
Still need to update the Service class documentation.

Comment 11 Chris Goldthorpe 2011-01-21 15:28:16 EST

This is not going to be ready for M5, changing milestone to 3.7M6.

Comment 12 Snehasish Paul 2011-02-03 14:24:00 EST

Created attachment 188265 [details]
Public API services with docs

Comment 13 Chris Goldthorpe 2011-02-08 17:34:22 EST

I just applied the most recent patch to I20110127-2034 on Windows and ran all the UA tests. I saw 67 errors and 10 failures. I tried on Linux also and saw 13 errors, no failures.

Here is a typical error:

org.eclipse.ua.tests.help.remote.TocServletTest
testTocServletContainsUAToc(org.eclipse.ua.tests.help.remote.TocServletTest)
org.xml.sax.SAXParseException: Invalid byte 2 of 4-byte UTF-8 sequence.

	at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)

	at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)

	at org.eclipse.ua.tests.help.remote.TocServletTest.getTocContributions(TocServletTest.java:122)

I can't figure out why this is happening, maybe it is a side effect of
Bug 336544 - [regression][compiler] Source flagged as dead code incorrectly.

Also ( far less important ) I see the word "valid" printed to the console, I think by SchemaValidator.java.

Comment 14 Snehasish Paul 2011-02-10 08:38:26 EST

Created attachment 188681 [details]
Public API services with UTF-8 byte error fix

In servlet's, we need to set the character encoding to UTF-8 before calling the response.getWriter(), else default charset = "ISO-8859-1" is set for the writer.

Comment 15 Chris Goldthorpe 2011-02-10 17:06:07 EST

I'm still seeing 54 errors and 10 failures on Windows XP with your latest patch.

Comment 16 Snehasish Paul 2011-02-17 11:45:41 EST

Hi,
I tried the patch with a fresh copy of Eclipse 3.7M5. This is working fine, runs 771 testcases with 0 error and 0 warning.

Comment 17 Chris Austin 2011-02-17 11:59:02 EST

I still see the errors in my environment - Snehasish, can you download the latest I Build and try it?

Comment 18 Chris Goldthorpe 2011-02-18 16:47:58 EST

I think I know what is happening. I am now seeing only two failing tests after having closed all of the projects in my workspace that are not part of Eclipse. The tests are making assumptions about such things as how many books exist in the table of contents, these assumptions are not valid because it is the expectation that the tests will continue to pass if more bundles are added - we need to be able to run these tests not just in the Eclipse SDK but also in a variety of Eclipse based products.

Comment 19 Chris Austin 2011-03-02 16:07:53 EST

Created attachment 190202 [details]
Final Patch

Fixed a copyright, added validatedServlet to platform.doc, and removed the 2 failing tests (see Bug 338732).

Comment 20 Chris Austin 2011-03-02 16:08:43 EST

Patch committed to HEAD.

Comment 21 Snehasish Paul 2011-05-10 06:36:46 EDT

Created attachment 195192 [details]
JSON fixes

Contains fixes for generation of corrupt JSON.
Also JSON support for Topic numeric path.

Comment 22 Snehasish Paul 2011-05-10 06:49:22 EDT

Sometimes JSON data values contains a value with some special characters, which are a part of JSON structure. Then client JSON parser not able to read JSON correctly.
Now all value field's are encode using the JAVA encoding:
java.net.URLEncoder.encode(val, "UTF-8")

All web Client should decode the values while reading JSON data using the custom JavaScript decode method:

decode = function(str) {
	str = '' + str;
	str = str.replace(/\+/g, " ");
	str = str.replace(/%7E/g, "~");
	str = str.replace(/%21/g, "!");
	str = str.replace(/%27/g, "'");
	str = str.replace(/%28/g, "(");
	str = str.replace(/%29/g, ")");
	
	return decodeURIComponent(str);
}

And Java client should use:
java.net.URLDecoder.decode(val, "UTF-8")

Also updated the code to create JSON for Topic numeric path xml data.

Comment 23 Chris Austin 2011-05-10 14:20:32 EDT

(In reply to comment #22)
> Sometimes JSON data values contains a value with some special characters, which
> are a part of JSON structure. Then client JSON parser not able to read JSON
> correctly.

This is a new issue.  Tracking in Bug 345309.