Re: CCSID of objects in IFS

[Scott Klement <midrange-l@xxxxxxxxxxxxxxxx>]

Hi Pete,

Just to be sure we're on the same page...    My understanding is that 
you are FTPing a "tar file" (aka a "tarball" in Unix-speak).  Is that 
correct?

If so -- the CCSID of the FTP is completely irrelevant.

Sure, the TAR file itself will get a CCSID assigned by FTP -- but that 
CCSID will be ignored, since TAR files are binary files, they don't get 
translated when the TAR utility opens them, so the CCSID assigned by FTP 
is ignored.

The "Tar" utility is a Unix utility, and knows nothing of CCSIDs. 
Therefore, when the contents of the tar file are extracted, and TAR 
creates new disk objects, the objects get a "default" CCSID.  If you set 
 QIBM_CCSID to 819 before running TAR, then the new disk objects will 
get a CCSID of 819.  If you don't, they'll get the default CCSID for 
your job (which, according to your screen shots, is 37).

The fact that your sysval is set to 65535 is not related to the problem 
at hand -- though I agree that 65535 is not ideal.

I don't understand the message
com.ibm.db2.jdbc.app.DB2JDBCException: CCSID value is not valid.

This appears to be connecting to a database somewhere (it's a JDBC 
message!).  I don't see what that has to do with the files you're 
extracting from your TARball...  Are those actually database files?  Are 
you running the AIX version of DB2 in PASE or something?  Otherwise, I 
don't see how the CCSID of the IFS files can be related to what you're 
doing...  The i5/OS DB2 doesn't read IFS files, it reads 
physical/logical files...


Pete Helgren wrote:
> Ok. Based on the feedback so far, it sounds like I have a CCSID issue 
> but I am still uncertain what the "fix" is.  I understand that the files 
> get tagged with certain CCSID's depending upon their origin, in this 
> case FTP probably set the files to 819.
>
> Specifically, I am looking for a suggestion on how to eliminate the 
> errors I get on some of the files that cannot be read, either by 
> changing the CCSID of the files in the IFS so I can run the scripts 
> within QSH or changing the job so it is "happy" with the CCSID of the 
> files it is reading.
>
> A bit more information might be helpful.  I am starting a session using 
> the QSH command.  I set the path, classpath and the current directory 
> and then type in a command which I assume is a shell command. In this 
> case, there is also some java being invoked and I am using a JDBC driver 
> (native).  It gives me the following error:
>
> com.ibm.db2.jdbc.app.DB2JDBCException: CCSID value is not valid.
>
> So that confirms that CCSID is at the bottem of all this.  Of course it 
> isn't nice enough to tell me WHAT doesn't have the correct value (the 
> jar, the class, the shell script, the data files, uh what?)
>
> So, I guess I am back to the original question.  Given the fact that the 
> files are CCSID 819 and it looks like the system value is set to 65535 
> (which Lukas doesn't) like.  The job attributes are
>
> Coded character set identifier  . . . . . . . . . :   65535
> Default coded character set identifier  . . . . . :   37  
>
> The user profile has:
>
> Coded character set ID . . . . .   *SYSVAL   
>
> Should I use the QIBM_CCSID environment variable to 819 as well?
>
> I feel I am on a slippery slope here.  I haven't really run into a 
> problem before running commands within QSH and have CCSID issues crop 
> up.  Perhaps I have been lucky but it seems like I have done something 
> fundamentally different in downloading extracting these files this time 
> around an this CCSID issue is now coming up.
>
>
> Pete
>
>
> Scott Klement wrote:
> > Not exactly sure what you're looking for, but...   think of a CCSID as a 
> > label.   If you think of a filing cabinet in an office, you put a little 
> > label on each of your files that tells what's in it.   The CCSID works 
> > much the same way, it identifies the character set of the data that's 
> > stored in a file.
> >
> > Programs can then read this CCSID from the "label" (i.e. object 
> > description) of the file to determine which character set the data is 
> > stored in, and then software can translate the data (or not) as 
> > appropriate for whatever that software does.
> >
> > Like the labels you use in your filing cabinet, it's possible to 
> > mistakenly put the wrong value on the file.  If the data in a file is 
> > ASCII, and you stick a label on it that says "EBCDIC", that's going to 
> > confuse the program that reads it.  If a program needs ASCII and the 
> > file says "EBCDIC", then it will take each byte of the file and run it 
> > through an EBCDIC->ASCII translation table.  If the data wasn't really 
> > EBCDIC to begin with, that'll result in gibberish.   Likewise, if the 
> > data was actually EBCDIC, but someone marked it as ASCII, the system 
> > might not try to translate it, resulting in gibberish...
> >
> > Obviously the best policy is to mark files with the correct CCSID for 
> > the data that's stored inside them.   But, when data is transferred from 
> > another source (another computer, etc) this might not be so easy. 
> > Typically it's the data that's transferred, and not the object 
> > description, so the system has to pick a default CCSID to put on the 
> > file.  That can be trouble...
> >
> > It sounds to me like you have your default set to 37 (which is a flavor 
> > of EBCDIC, and is the one we use in the USA most of the time).  But the 
> > data in your files is ASCII data.  When you untar your objects, there's 
> > no CCSID stored in the tarball (tar is designed for Unix systems, which 
> > don't really have the notion of CCSID, at least not as an external 
> > marking on the file).  So the system is assigning the default one of 37, 
> > which is wrong for this particular application.
> >
> > By default, QShell will create new files using the job's CCSID (which is 
> > probably where the 37 is coming from) but you can set the QIBM_CCSID 
> > environment variable to control this.  More info here:
> > http://publib.boulder.ibm.com/infocenter/iseries/v5r4/topic/rzahz/variable.htm
> >
> > As for the other system where things worked as you wanted them to... I 
> > have a feeling that PASE was used instead of QShell... just a stab in 
> > the dark, but PASE typically assumes everything is ASCII, since it's 
> > emulating an ASCII system, whereas QShell acknowledges EBCDIC as a 
> > viable alternative :)
> >
> >
> > Pete Helgren wrote:
> >   
> > > I am a little confused about what CCSID objects in the IFS should have 
> > > and how the CCSID is set.
> > >
> > > I was running some scripts in QSH and noticed some syntax errors when I 
> > > ran the scripts on our 520 at V5R4M0 vs the 270 at V5R3M0.  On both 
> > > machines I had FTP'd a tar file and extracted the tar file and then ran 
> > > the scripts.  On the 520 the CCSID of the files extracted is 37.  On the 
> > > 270, the files have the CCSID of 819.  I am not sure why there is a 
> > > difference and where the CCSID would be set.
> > >
> > > Any pointers here would be helpful.
> > >
> > > Pete Helgren
> > >     
> >