Re: Problems with YAJL and/or SoapUI -- WEB400

Jon,

A few observations...

1) You are working with JSON data, *not* application/x-www-form-urlencoded. The content-type for JSON data is application/json.

application/x-www-form-urlencoded is data like you would put in a web URL from HTML form/input tags. I'm sure you've worked with it before, it looks something like this:

custno=1234&name=Jon%20Paris

JSON data, by contrast, looks like this:

{ "custno": 1234, "name": "Jon Paris" }

Not the same, right? Make sure you are using application/json.

2) YAJL did *not* tell you 'unsupported content type'. YAJL just assumes any data passed to it is in JSON format, and does not look at the content type at all. This message would've been from SoapUI.

3) If you had to tell YAJL that the data was in EBCDIC, it may be because you specified application/x-www-form-urlencoded since Apache will interpret that and translate it to EBCDIC. So if you sent data that is really JSON but said it was "form-urlencoded' it might have made this work as EBCDIC -- but this is a really bad way to do things that may cause problems down the road. You are confusing Apache and your web service client.

4) The default CgiConvMode is "ebcdic". I mention this because some posts suggest it might be mixed or something else.

5) SoapUI has different tabs that view the response as XML, JSON, RAW, etc. If your data is JSON, it's not going to be able to interpret it as XML, HTML, etc. You say "Goodness knows why HTML didn't work"... but... doesn't it make sense that JSON data can't be interpreted as HTML?

Pretty much all of your mistakes seem to stem from telling the computer "it is X" when it really is "Y".

This isn't about HTTPAPI, it's about JSON/YAJL.
The data isn't EBCDIC, it's UTF-8.
The data isn't a URL-encoded form, it is JSON
The data isn't HTML, it is JSON

When working with people, you can often get away with mislabeling things because people will figure it out. If you were to go to a party with a name tag that says "Hello, my name is Susan" it would not be a problem, people would look at you and know you're not Susan. The human brain is really good at looking at something and determining what it is.

A computer doesn't think that way. Everything is a stream of binary ones and zeroes... it's up to you to tell it how to interpret that stream. It isn't good at figuring it out for you. :-)

-SK

On 6/26/2017 4:29 PM, Jon Paris wrote:

Not really.

I have to believe that I screwed it up when I first tried to fix the "Why do I need to tell it EBCDIC when it is UTF-8" issue. Only thing I can think is that I subsequently misread the SoapUI response apparently not noticing that there was in fact real data behind the apparent error message. From there I ended up just trying many, many combinations all of which apparently failed in different ways. It took Henrik's program to help me realize that I was getting the right answers and made me look at the SoapUI response again.

This is what you get when you foolishly decide to leap into (relatively) new areas on a Sunday when there's nobody around to ask questions.

Again thanks to you and all the others who helped. Sometimes being told that you are already doing the right thing is what you need to reinvestigate your assumptions!

Jon Paris

www.partner400.com
www.SystemiDeveloper.com

On Jun 26, 2017, at 2:41 PM, Kevin Turner <kevin.turner@xxxxxxxxxxxxxx> wrote:

But you still don't know why it was working before (when converting to EBCDIC) and then stopped? Or do you? Or did it never stop working? :)

________________________________________
From: WEB400 [web400-bounces@xxxxxxxxxxxx] on behalf of Jon Paris [jon.paris@xxxxxxxxxxxxxx]
Sent: 26 June 2017 19:36
To: Web400@Midrange. Web400
Subject: Re: [WEB400] Problems with YAJL and/or SoapUI

[External Email]
________________________________

I have decided to start believing in fairies.

In looking for an indication of how to specify anticipated content type I noticed that I had inadvertently clicked on the HTML view tab in SoapUI. When I clicked on Raw I could see the response. Goodness knows why the HTML shows an error but ...

I know that I had looked at that setting before but I guess I had tried so many different combinations I just missed seeing the results when the combo was correct.

So it looks as if the change to application/json worked - but I just didn't notice it in the 487th iteration of the test cycles.

Many, many thanks to all who contributed to this resolution. Nice to know I was completely nuts.

Particular thanks to Henrik who, although he was answering the wrong question at the time <grin>, supplied me with the quick example that allowed me to test without SoapUI without spending time building the code. Having an example that I knew worked gave me the confidence that any errors remaining were outside the scope of that consumer code.

Jon Paris

www.partner400.com
www.SystemiDeveloper.com

On Jun 26, 2017, at 2:25 PM, Kevin Turner <kevin.turner@xxxxxxxxxxxxxx> wrote:

So SoapUI is not sending the data in UTF-8?

I am no expert on that tool but there must be a way to force that. Example: https://stackoverflow.com/questions/35747109/where-do-i-set-character-encoding-in-soapui-preferences

________________________________________
From: WEB400 [web400-bounces@xxxxxxxxxxxx] on behalf of Jon Paris [jon.paris@xxxxxxxxxxxxxx]
Sent: 26 June 2017 19:19
To: Web400@Midrange. Web400
Subject: Re: [WEB400] Problems with YAJL and/or SoapUI

[External Email]
________________________________

Tried telling SoapUI that Brad, and that is what Henrik's code does.

Interestingly - now that I have modified my web service code to say that the input is UTF-8 it all works with Henrik's code. I had forgotten what state I left that in on the last go-round.

Sadly SoapUI still returns "unsupported content-type [application/json; charset=utf-8]" - so while I have a way of testing the web service again I still cannot get SoapUI to work.

weird.

Jon Paris

www.partner400.com
www.SystemiDeveloper.com

On Jun 26, 2017, at 1:03 PM, Bradley Stone <bvstone@xxxxxxxxx> wrote:

The content-type should be application/json. When it is that then the
Apache server won't do any conversions on it (or at least it
shouldn't).

Most other content types will do ASCII to EBCDIC conversions automatically.

Bradley V. Stone
www.bvstools.com
MAILTOOL Benefit #2: The ability to specify a "From" and/or "Reply To"
email address!

On Mon, Jun 26, 2017 at 11:37 AM, Henrik Rützou <hr@xxxxxxxxxxxx> wrote:

There is no Apache involved in HTTPAPI!

On Mon, Jun 26, 2017 at 6:35 PM, Kevin Turner <kevin.turner@xxxxxxxxxxxxxx>
wrote:

Just a thought (I admit I have only scanned your post rather than read it
in detail).

You don't appear to have specified cgiConvMode in your Apache config, so I
think it will default to MIXED

have you tried forcing the conversion mode and CCSID situation?

Example

DefaultNetCCSID 819
CGIJobCCSID 285
CGIConvMode EBCDIC

[https://www.netcracker.com/assets/img/netcracker-social-final.png] ƕ
________________________________________
From: WEB400 [web400-bounces@xxxxxxxxxxxx] on behalf of Jon Paris [
jon.paris@xxxxxxxxxxxxxx]
Sent: 26 June 2017 15:52
To: Web400@Midrange. Com
Subject: [WEB400] Problems with YAJL and/or SoapUI

[External Email]
________________________________

Posted this on HTTPAPI list but no takers so far. Hoping someone here can
offer a suggestion or three.

I have been testing a JSON based web service with SoapUI and all was

going well.

_But_ in order to make my code work initially I had had to use:

root = yajl_stdin_load_tree ( *Off: errMsg );

In other words I had to tell YAJL that my data was in EBCDIC. This

didn't make sense to me because I thought I was sending UTF8 but I needed
it working and thought that resolution could wait for another day. Got the
whole service working beautifully.

But two things happened

1) "another day" has arrived and

2) A problem with SoapUI has meant my request configuration was not

saved. Tried to set it up again and now I cannot get a single *%^$ thing
working.

If I tell SoapUI that the data is "application/x-www-form-urlencoded" -

which I am pretty certain is what it was set as - then I get a message back
from YAJL saying "unsupported content-type [application/json;
charset=utf-8]"

OK - data _is_ UTF8 so I changed the *Off in the load tree to *On. But

now YAJL simply blows up during the tree load.

I have tried every combination of content setting and *On/*Off

combination I can think of and nothing is working. Even if I get past the
load tree blowing up then YAJL blows up subsequently when trying to locate
the first node.

Although I did play with the Apache config (trying to fix the

UTF8/EBCDIC issue) I'm certain it is now back where it was and, other than
the noted *On/*Off changes mentioned above, the program has not been
changed from the working version.

Apache config is straightforward as set by the wizard with changes only

to the ScriptAlias - it looks like this:

1 # Configuration originally created by Create HTTP Server wizard

on Mon Jan 23 16:49:35 EST 2012

2 Listen *:1241
3 DocumentRoot /www/jonrest/htdocs
4 TraceEnable Off
5 Options +ExecCGI -FollowSymLinks -SymLinksIfOwnerMatch

-Includes -Indexes -MultiViews

6 LogFormat "%h %T %l %u %t \"%r\" %>s %b \"%{Referer}i\"

\"%{User-Agent}i\"" combined

7 LogFormat "%{Cookie}n \"%r\" %t" cookie
8 LogFormat "%{User-agent}i" agent
9 LogFormat "%{Referer}i -> %U" referer
10 LogFormat "%h %l %u %t \"%r\" %>s %b" common
11 CustomLog logs/access_log combined
12 LogMaint logs/access_log 7 0
13 LogMaint logs/error_log 7 0
14 SetEnvIf "User-Agent" "Mozilla/2" nokeepalive
15 SetEnvIf "User-Agent" "JDK/1\.0" force-response-1.0
16 SetEnvIf "User-Agent" "Java/1\.0" force-response-1.0
17 SetEnvIf "User-Agent" "RealPlayer 4\.0" force-response-1.0
18 SetEnvIf "User-Agent" "MSIE 4\.0b2;" nokeepalive
19 SetEnvIf "User-Agent" "MSIE 4\.0b2;" force-response-1.0
20 MaxCGIJobs 3
21 MaxThreadedCGIJobs 3
22
23 ScriptAlias /SinglePartRequest /qsys.lib/xxxxxx.lib/

Singlepart.pgm

24 <Directory /qsys.lib/xxxxxx.lib >
25 Order Allow,Deny
26 Allow From All
27 SetEnv QIBM_CGI_LIBRARY_LIST "YAJL"
28 </Directory>
29
30 <Directory />
31 Order Deny,Allow
32 Deny From all
33 </Directory>
34 <Directory /www/jonrest/htdocs>
35 Order Allow,Deny
36 Allow From all
37 </Directory>
38

Hopefully someone with more experience in this area can point me in the

right direction because right now I'm clueless.

Jon Paris

www.partner400.com <http://www.partner400.com/>
www.SystemiDeveloper.com

--
This is the Web Enabling the IBM i (AS/400 and iSeries) (WEB400) mailing
list
To post a message email: WEB400@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/web400
or email: WEB400-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/web400.

________________________________
The information transmitted herein is intended only for the person or
entity to which it is addressed and may contain confidential, proprietary
and/or privileged material. Any review, retransmission, dissemination or
other use of, or taking of any action in reliance upon, this information by
persons or entities other than the intended recipient is prohibited. If you
received this in error, please contact the sender and delete the material
from any computer.
--
This is the Web Enabling the IBM i (AS/400 and iSeries) (WEB400) mailing
list
To post a message email: WEB400@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/web400
or email: WEB400-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/web400.

--
Regards,
Henrik Rützou

http://powerEXT.com <http://powerext.com/>
--
This is the Web Enabling the IBM i (AS/400 and iSeries) (WEB400) mailing list
To post a message email: WEB400@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/web400
or email: WEB400-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/web400.

--
This is the Web Enabling the IBM i (AS/400 and iSeries) (WEB400) mailing list
To post a message email: WEB400@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/web400
or email: WEB400-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/web400.

--
This is the Web Enabling the IBM i (AS/400 and iSeries) (WEB400) mailing list
To post a message email: WEB400@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/web400
or email: WEB400-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/web400.

________________________________
The information transmitted herein is intended only for the person or entity to which it is addressed and may contain confidential, proprietary and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.
--
This is the Web Enabling the IBM i (AS/400 and iSeries) (WEB400) mailing list
To post a message email: WEB400@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/web400
or email: WEB400-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/web400.

--
This is the Web Enabling the IBM i (AS/400 and iSeries) (WEB400) mailing list
To post a message email: WEB400@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/web400
or email: WEB400-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/web400.

________________________________
The information transmitted herein is intended only for the person or entity to which it is addressed and may contain confidential, proprietary and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.
--
This is the Web Enabling the IBM i (AS/400 and iSeries) (WEB400) mailing list
To post a message email: WEB400@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/web400
or email: WEB400-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/web400.