Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.
Bug 359643 - non-US characters not processed from Web Service dataset
Summary: non-US characters not processed from Web Service dataset
Status: RESOLVED FIXED
Alias: None
Product: z_Archived
Classification: Eclipse Foundation
Component: BIRT (show other bugs)
Version: 3.7.2   Edit
Hardware: PC Windows 7
: P3 normal (vote)
Target Milestone: 4.2.0   Edit
Assignee: Birt-DataAccess CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-10-01 16:55 EDT by Nicklas Karlsson CLA
Modified: 2012-08-22 04:55 EDT (History)
2 users (show)

See Also:


Attachments
steps to reproduce (4.20 KB, application/x-zip-compressed)
2011-10-01 16:59 EDT, Nicklas Karlsson CLA
no flags Details
return the charactors (59.15 KB, image/jpeg)
2011-12-31 03:46 EST, Xiaoying Gu CLA
no flags Details
report (13.07 KB, application/octet-stream)
2012-05-23 06:35 EDT, Nicklas Karlsson CLA
no flags Details
data-providing war (3.47 KB, application/octet-stream)
2012-05-23 06:36 EDT, Nicklas Karlsson CLA
no flags Details
source for test (2.68 KB, application/octet-stream)
2012-05-23 06:37 EDT, Nicklas Karlsson CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Nicklas Karlsson CLA 2011-10-01 16:55:17 EDT
Build Identifier: 

Given a Web Service that returns data of non-US characters, a BIRT datasource against that service will stop processing rows as soon a first non-US char is processed.

Reproducible: Always

Steps to Reproduce:
1. Create Web Service returning non-US data
2. Create a Data Source and Data Set against service
3. Behold no data
Comment 1 Nicklas Karlsson CLA 2011-10-01 16:59:15 EDT
Created attachment 204422 [details]
steps to reproduce

Birt.java - a sample WebService returning good and bad data
Birt_wsdl.xml - sample wsdl from the service
good_request.xml - sample request XML from SoapUI call 
good_response.xml - sample response XML from SoapUI call 
bad_request.xml - sample request XML from SoapUI call
bad_response.xml - sample response XML from SoapUI call
test.rptdesign - report showing failure to show bad data.

SoapUI shows correct data for both good and bad
Comment 2 Maggie Shen CLA 2011-11-14 01:04:15 EST
Hi,

I tried a web service (ttp://webservice.webxml.com.cn/WebServices/WeatherWebService.asmx?wsdl) returning Chinese characters without error.I will try your given environment later.
Comment 3 Xiaoying Gu CLA 2011-12-31 03:46:27 EST
Created attachment 208887 [details]
return the charactors
Comment 4 Xiaoying Gu CLA 2011-12-31 03:48:49 EST
hi Nicklas Karlsson 

In the bird version 3.7.2 I had reproduced the steps as you prividing for.but it didn't occured. please refer to the attrached.
Comment 5 Nicklas Karlsson CLA 2012-01-02 04:29:39 EST
Is there some downloadable artifact for 3.7.2? I see no stable builds in the list, do I have to build from source?
Comment 6 Xiaoying Gu CLA 2012-01-04 03:46:09 EST
(In reply to comment #5)
> Is there some downloadable artifact for 3.7.2? I see no stable builds in the
> list, do I have to build from source?

Please wait for the upcoming 3.7.2 RC1 build to verify
Comment 7 Nicklas Karlsson CLA 2012-05-23 06:33:28 EDT
The issue remains for me. Given a WebService WAR (attached) on a JBoss 7.1.0.Final AS I see the following when running a soapUI 3.6.1:

13:15:56,148 INFO  [org.apache.cxf.interceptor.LoggingInInterceptor] (http--0.0.0.0-8080-1) Inbound Message
----------------------------
ID: 2
Address: http://localhost:8080/BirtData/DataProvider
Encoding: UTF-8
Http-Method: POST
Content-Type: text/xml;charset=UTF-8
Headers: {accept-encoding=[gzip,deflate], Content-Length=[208], content-type=[text/xml;charset=UTF-8], host=[localhost:8080], SOAPAction=[""], user-agent=[Jakarta Commons-HttpClient/3.1]}
Payload: <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:nik="http://nik.fi/">
   <soapenv:Header/>
   <soapenv:Body>
      <nik:goodData/>
   </soapenv:Body>
</soapenv:Envelope>
--------------------------------------
13:15:56,237 INFO  [org.apache.cxf.interceptor.LoggingOutInterceptor] (http--0.0.0.0-8080-1) Outbound Message
---------------------------
ID: 2
Encoding: UTF-8
Content-Type: text/xml
Headers: {}
Payload: <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><ns2:goodDataResponse xmlns:ns2="http://nik.fi/"><return><item>aao</item></return></ns2:goodDataResponse></soap:Body></soap:Envelope>
--------------------------------------
13:16:03,884 INFO  [org.apache.cxf.interceptor.LoggingInInterceptor] (http--0.0.0.0-8080-1) Inbound Message
----------------------------
ID: 3
Address: http://localhost:8080/BirtData/DataProvider
Encoding: UTF-8
Http-Method: POST
Content-Type: text/xml;charset=UTF-8
Headers: {accept-encoding=[gzip,deflate], Content-Length=[207], content-type=[text/xml;charset=UTF-8], host=[localhost:8080], SOAPAction=[""], user-agent=[Jakarta Commons-HttpClient/3.1]}
Payload: <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:nik="http://nik.fi/">
   <soapenv:Header/>
   <soapenv:Body>
      <nik:badData/>
   </soapenv:Body>
</soapenv:Envelope>
--------------------------------------
13:16:03,895 INFO  [org.apache.cxf.interceptor.LoggingOutInterceptor] (http--0.0.0.0-8080-1) Outbound Message
---------------------------
ID: 3
Encoding: UTF-8
Content-Type: text/xml
Headers: {}
Payload: <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><ns2:badDataResponse xmlns:ns2="http://nik.fi/"><return><item>åäö</item></return></ns2:badDataResponse></soap:Body></soap:Envelope>
--------------------------------------

The "goodData" provides "aao" and the "badData" provides "åäö". The endpoint can be proven to provide this (correct) output when accessed through an Apache HttpClient call (using both UTF-8 and ISO-8859-1) encoding.

When I try running the BIRT report (attached). I only get the "goodData" output in the report and in the logs I see

13:29:32,699 INFO  [org.apache.cxf.interceptor.LoggingInInterceptor] (http--0.0.0.0-8080-1) Inbound Message
----------------------------
ID: 9
Address: http://fikarlsnic01:8080/BirtData/DataProvider
Encoding: ISO-8859-1
Http-Method: POST
Content-Type: text/xml
Headers: {Accept=[text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2], connection=[Close], Content-Length=[370], content-type=[text/xml], host=[fikarlsnic01:8080], SOAPAction=[], user-agent=[Java/1.6.0_20]}
Payload: <?xml version="1.0"?>
<SOAP-ENV:Envelope
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
	<SOAP-ENV:Body>
		<m:goodData xmlns:m="http://nik.fi/">
		</m:goodData>
	</SOAP-ENV:Body>
</SOAP-ENV:Envelope>
--------------------------------------
13:29:32,704 INFO  [org.apache.cxf.interceptor.LoggingOutInterceptor] (http--0.0.0.0-8080-1) Outbound Message
---------------------------
ID: 9
Encoding: ISO-8859-1
Content-Type: text/xml
Headers: {}
Payload: <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><ns2:goodDataResponse xmlns:ns2="http://nik.fi/"><return><item>aao</item></return></ns2:goodDataResponse></soap:Body></soap:Envelope>
--------------------------------------
13:29:32,725 INFO  [org.apache.cxf.interceptor.LoggingInInterceptor] (http--0.0.0.0-8080-2) Inbound Message
----------------------------
ID: 10
Address: http://fikarlsnic01:8080/BirtData/DataProvider
Encoding: ISO-8859-1
Http-Method: POST
Content-Type: text/xml
Headers: {Accept=[text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2], connection=[Close], Content-Length=[368], content-type=[text/xml], host=[fikarlsnic01:8080], SOAPAction=[], user-agent=[Java/1.6.0_20]}
Payload: <?xml version="1.0"?>
<SOAP-ENV:Envelope
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
	<SOAP-ENV:Body>
		<m:badData xmlns:m="http://nik.fi/">
		</m:badData>
	</SOAP-ENV:Body>
</SOAP-ENV:Envelope>
--------------------------------------
13:29:32,728 INFO  [org.apache.cxf.interceptor.LoggingOutInterceptor] (http--0.0.0.0-8080-2) Outbound Message
---------------------------
ID: 10
Encoding: ISO-8859-1
Content-Type: text/xml
Headers: {}
Payload: <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><ns2:badDataResponse xmlns:ns2="http://nik.fi/"><return><item>???</item></return></ns2:badDataResponse></soap:Body></soap:Envelope>
--------------------------------------

and no data is displayed for the "badData"
Comment 8 Nicklas Karlsson CLA 2012-05-23 06:35:11 EDT
Created attachment 216109 [details]
report
Comment 9 Nicklas Karlsson CLA 2012-05-23 06:36:28 EDT
Created attachment 216111 [details]
data-providing war

Deploy on a JBoss 7.1.x
Comment 10 Nicklas Karlsson CLA 2012-05-23 06:37:13 EDT
Created attachment 216113 [details]
source for test
Comment 11 Nicklas Karlsson CLA 2012-05-23 06:38:39 EDT
I'll have to check my log settings, yesterday I was seeing the "invalid byte 2 of 3 byte..." encoding error that came with the report WS call
Comment 12 Nicklas Karlsson CLA 2012-05-24 03:18:31 EDT
Wonder if those ??? come from the encoding of the log file? In any case, the issue remains. And when using httpClient and storing the result myself, I see the WS providing the correct åäö regardeless of UTF-8/ISO-8859-1 encoding
Comment 13 Nicklas Karlsson CLA 2012-06-06 06:04:31 EDT
Is there any additional information I can provide you with?
Comment 14 Nicklas Karlsson CLA 2012-08-16 08:31:33 EDT
There are indications that this issue is no longer present in the 4.2.0 version
Comment 15 Xiaoying Gu CLA 2012-08-22 04:55:44 EDT
Set as fixed based on comment #14