Re: CPYTOIMPF and accented characters

[Mohammad Tanveer <surgum@xxxxxxxxx>]

I tried the UTF-8 with BOM but still it didn't work as per the link.. When
I open the file in Notepad ++ I see my encoding is set to UTF-8 BOM.  as
you can see from below screen shot characters are translating properly but
not the accented one.. they appear as plain text Å appears as $.

[image: Inline image 1]


On Wed, Nov 2, 2016 at 9:02 AM, Justin Dearing <zippy1981@xxxxxxxxx> wrote:

> Mohammad,
>
> You really need a hex editor when fighting encodings. There are two basic
> facts to consider. What is the actual byte in that stream at that point in
> the file, and what is the encoding the text editor is either reading from
> the BOM or assuming.
>
> It looks like CPYTOIMPF does not add a BOM to the begining of the file, but
> its not terrible hard to do, but a little hacky.
> http://www-01.ibm.com/support/docview.wss?uid=nas8N1011066
>
> But get a hex editor plugin for notepad++. Thats the best way to debug
> unicode issues.
>
> BTW, if you want a text editor that can read EBCDIC, I recommend the one
> built into the free and open source orthodox file manager farmanager. Its
> developed by russians, but the english internationalization is great.
>
> Justin Dearing
>
> On Wed, Nov 2, 2016 at 11:39 AM Mohammad Tanveer <surgum@xxxxxxxxx> wrote:
>
> > IBM I version 5770SS1 V7R2M0
> >
> > tried multiple combinations
> > CCSID of the DB2 Column (tried with 37, 278, 1143)
> > CCSID of the IFS File (278, 1251, 1143)
> > CCSID of the job is 65535
> >
> >
> > On Wed, Nov 2, 2016 at 8:23 AM, Mohammad Tanveer <surgum@xxxxxxxxx>
> wrote:
> > > I tried FTP, SFTP and simple drag and drop of IFS file to PC.  same
> > > behavior all the time.
> > >
> > > On Wed, Nov 2, 2016 at 6:52 AM, Rob Berendt <rob@xxxxxxxxx> wrote:
> > >
> > > > How are you "moving" the file?  Some methods are more friendly than
> > others
> > > > in preserving such characters.  FTP is generally recommended as the
> > > > safest.  I suspect you're doing that because you mentioned ASC and
> BIN.
> > > > Rob Berendt
> > > > --
> > > > IBM Certified System Administrator - IBM i 6.1
> > > > Group Dekko
> > > > Dept 1600
> > > > Mail to:  2505 Dekko Drive
> > > >           Garrett, IN 46738
> > > > Ship to:  Dock 108
> > > >           6928N 400E
> > > >           Kendallville, IN 46755
> > > > http://www.dekko.com
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > From:   Mohammad Tanveer <surgum@xxxxxxxxx>
> > > > To:     Midrange Systems Technical Discussion <
> midrange-l@xxxxxxxxxxxx>
> > > > Date:   11/02/2016 09:48 AM
> > > > Subject:        Re: CPYTOIMPF and accented characters
> > > > Sent by:        "MIDRANGE-L" <midrange-l-bounces@xxxxxxxxxxxx>
> > > >
> > > >
> > > >
> > > > I know that on IFS file has correct character code as I can see the
> data
> > > > on
> > > > AS/400 correctly.  Problem is only if I move the CSV file to PC
> accented
> > > > character translation is getting lost somehow...
> > > >
> > > > On Wed, Nov 2, 2016 at 6:45 AM, Mohammad Tanveer <surgum@xxxxxxxxx>
> > > > wrote:
> > > >
> > > > > I tried both.. in Notepad++ I see special codes in front of almost
> all
> > > > > SWE/DEN characters when I switch the encoding to UTF-8 but not for
> > > > accented
> > > > > characters that's why I am thinking for some reason accented
> > characters
> > > > are
> > > > > not converted properly or translation is not happening.
> > > > >
> > > > > On Wed, Nov 2, 2016 at 6:34 AM, Luis Rodriguez <luisro58@xxxxxxxxx>
> > > > wrote:
> > > > > > Are you using the "standard" Windows Notepad? What happens if yu
> open
> > > > that
> > > > > > file using an app like Notepad++ or Word? It could be a problem in
> > how
> > > > > > Notepad is opening the file...
> > > > > >
> > > > > > HTH,
> > > > > >
> > > > > > Luis
> > > > > >
> > > > > >
> > > > > > On Wed, Nov 2, 2016 at 9:29 AM Mohammad Tanveer <surgum@xxxxxxxxx>
> > > > wrote:
> > > > > > > First I tried CCSID 277 it didn't work then I tried 278 and then
> a
> > > > > > number
> > > > > > > of others..  nothing works... as soon as I download the CSV to PC
> > > > from
> > > > > > IFS
> > > > > > > I don't see accented characters appearing in notepad.
> > > > > > >
> > > > > > > Regards
> > > > > > > .
> > > > > > >
> > > > > > > On Wed, Nov 2, 2016 at 6:05 AM, Luis Rodriguez <
> luisro58@xxxxxxxxx
> > > > > > wrote:
> > > > > > > > Are you sure about the CCSID? A quick Google search gave me 277
> > for
> > > > > > > > Danish...
> > > > > > > > Regards,
> > > > > > > >
> > > > > > > > Luis
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Nov 2, 2016 at 9:00 AM Mohammad Tanveer <
> > surgum@xxxxxxxxx>
> > > > > > > wrote:
> > > > > > > > > After creating a CSV file using CPYTOIMPF and CCSID 278 or
> > > > *PCASCII.
> > > > > > > > > It looks like accented characters (Danish characters like A
> > with
> > > > > > circle
> > > > > > > > on
> > > > > > > > > top *Å*) do show up in 5250 emulation when I use EDTF on IFS
> > file
> > > > > > but
> > > > > > > > they
> > > > > > > > > don't show up when I copy the file to PC and open it in text
> > > > editor
> > > > > > > with
> > > > > > > > > UTF-8 encoding.  It shows as $ sign.  Most of other Danish
> > > > > > characters
> > > > > > > > show
> > > > > > > > > up correctly in text file. I tried copying the file to PC in
> > > > binary
> > > > > > > > format
> > > > > > > > > and ASCII but no luck.
> > > > > > > > >
> > > > > > > > > Is it a bug or I cannot convert Accented characters using
> > > > CPYTOIMPF
> > > > > > for
> > > > > > > > > PC?  Other CCSID 278 characters are converting fine.
> > > > > > > > >
> > > > > > > > > Following output is in EDTF after changing the emulation
> > setting
> > > > to
> > > > > > > host
> > > > > > > > > code 037.  Line 5 shows $ instead of *Å*
> > > > > > > > >
> > > > > > > > >  Browse :
> > > > > > > > > /swe/SL600O.csv
> > > > > > > > >
> > > > > > > > >  Record :      18   of    1905 by  18
> > Column
> > > > :
> > > > > > > 1
> > > > > > > > > 225 by 131
> > > > > > > > >  Control
> > > > > > > > > :
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > ....+....1....+....2....+....3....+....4....+....5....+....
> > > > > > > > 6....+....7....+....8....+....9....+....0....+....1....+....
> 2..
> > > > > > > > > 1908932044     ,460000643478,MALENE HESSELDAL
> > > > > > > > > ,GUNDERSEN                     ,JERICHAUSGADE 3  ST. TV
> > > > > > > > > 0311732102     ,460000601288,KARINA
> > > > > > > > > ,AKSTRUP                       ,LILLEVANG 145
> > > > > > > > > 2805742472     ,460000741415,MAJ
> > > > > > > > > ,ANDKJÆR                       ,SOLSIKKEVÆNGET 22
> > > > > > > > > 9203064069 <(920)%20306-4069>     ,460000690554,EMILY
> > > > > > > > > ,MIKKELSEN                     ,ROTTARPVEJ 32
> > > > > > > > > 300368-0028    ,460000015491,GITTE
> > > > > > > > > ,PEDERSEN                      *,M$GEVEJ 44   *
> > > > > > > > > 110871-1228    ,460000503509,LOTTE
> > > > > > > > > ,ERRI                          ,KÆRBØLVEJ 52 A
> > > > > > > > > 0304712252     ,460000695645,METTE
> > > > > > > > > ,AUNBIRK                       ,HOLMELINSVEJ 3
> > > > > > > > > 1008762682     ,460000713579,NICOLE
> > > > > > > > > ,FOGET                         ,SOLVANGSVEJ 5B
> > > > > > > > > 0508691254     ,460000705806,FREYA CHALOTTE
> > > > > > > > > ,HOLM                          ,ØSTERGADE 9
> > > > > > > > > 0411742024     ,460000707761,PIA VIBEKE
> > > > > > > > > ,HAUSTED                       ,KAMPESMOSE 23
> > > > > > > > > 2203781132 <(220)%20378-1132>     ,460000673722,MARIE
> > > > > > > > > ,KJÆR                          ,TATERBAKKEVEJ 3
> > > > > > > > > 240259-2020    ,460000012598,LONE
> > > > > > > > > ,MARQUARDSEN                   ,FORSYTHIAVEJ 38
> > > > > > > > > 1204762526     ,460000699274,MALENE DALGAARD
> > > > > > > > > ,KROGH                         ,CENTRALG$RDSVEJ 121
> > > > > > > > > 2103572528 <(210)%20357-2528>     ,460000518039,DORRIT
> > > > > > > > > ,KRISTENSEN                    ,SKJOLDAGERVEJ 92  2 TH
> > > > > > > > > 0507380840     ,460000539850,KARIN
> > > > > > > > > ,SIMONSEN                      ,STRANDGADE 28 N HOU
> > > > > > > > > 1208520144     ,460000639066,ANNIE
> > > > > > > > > ,BEJART                        ,PRÆSTEGAARDSVEJ 5
> > > > > > > > > 2403741356 <(240)%20374-1356>     ,460000729054,KAREN
> > > > > > > > > ,EBBESEN                       ,BELLAHOJVEJ 14
> > > > > > > > > 1903802622     ,460000675467,VINNI MARIE         ,S.
> > > > > > > > > MADSEN                     ,ØSTKÆRVEJ 27
> > > > > > > > > 2604712478 <(260)%20471-2478>     ,460000701355,METTE
> > > > > > > > > ,OLESEN                        ,BYAGEREN 25
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > This is after switching the emulation to Host Code Page to
> 1143
> > > > you
> > > > > > can
> > > > > > > > see
> > > > > > > > > line 5 show correctly in EDT.. and I can copy paste this data
> > in
> > > > > > > notepad
> > > > > > > > as
> > > > > > > > > you can see PC shows it correctly.  Which means PC can
> > understand
> > > > > > this
> > > > > > > > > character in notepad... so why CPYTOIMPF is not able to
> convert
> > > > it
> > > > > > to
> > > > > > > > plain
> > > > > > > > > PC text using CCSID?
> > > > > > > > >
> > > > > > > > >  Browse :
> > > > > > > > > /swe/SL600O.csv
> > > > > > > > >
> > > > > > > > >  Record :      18   of    1905 by  18
> > Column
> > > > :
> > > > > > > 1
> > > > > > > > > 225 by 131
> > > > > > > > >  Control
> > > > > > > > > :
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > ....+....1....+....2....+....3....+....4....+....5....+....
> > > > > > > > 6....+....7....+....8....+....9....+....0....+....1....+....
> > > > > > 2....+....3.
> > > > > > > > > 1908932044     ,460000643478,MALENE HESSELDAL
> > > > > > > > > ,GUNDERSEN                     ,JERICHAUSGADE 3  ST.
> > > > > > > > > TV
> > > > > > > > > 0311732102     ,460000601288,KARINA
> > > > > > > > > ,AKSTRUP                       ,LILLEVANG
> > > > > > > > > 145
> > > > > > > > > 2805742472     ,460000741415,MAJ
> > > > > > > > > ,ANDKJÆR                       ,SOLSIKKEVÆNGET
> > > > > > > > > 22
> > > > > > > > > 9203064069 <(920)%20306-4069>     ,460000690554,EMILY
> > > > > > > > > ,MIKKELSEN                     ,ROTTARPVEJ
> > > > > > > > > 32
> > > > > > > > > 300368-0028    ,460000015491,GITTE
> > > > > > > > > ,PEDERSEN                      *,MÅGEVEJ 44   *
> > > > > > > > >
> > > > > > > > > 110871-1228    ,460000503509,LOTTE
> > > > > > > > > ,ERRI                          ,KÆRBØLVEJ 52
> > > > > > > > > A
> > > > > > > > > 0304712252     ,460000695645,METTE
> > > > > > > > > ,AUNBIRK                       ,HOLMELINSVEJ
> > > > > > > > > 3
> > > > > > > > > 1008762682     ,460000713579,NICOLE
> > > > > > > > > ,FOGET                         ,SOLVANGSVEJ
> > > > > > > > > 5B
> > > > > > > > > 0508691254     ,460000705806,FREYA CHALOTTE
> > > > > > > > > ,HOLM                          ,ØSTERGADE
> > > > > > > > > 9
> > > > > > > > > 0411742024     ,460000707761,PIA VIBEKE
> > > > > > > > > ,HAUSTED                       ,KAMPESMOSE
> > > > > > > > > 23
> > > > > > > > > 2203781132 <(220)%20378-1132>     ,460000673722,MARIE
> > > > > > > > > ,KJÆR                          ,TATERBAKKEVEJ
> > > > > > > > > 3
> > > > > > > > > 240259-2020    ,460000012598,LONE
> > > > > > > > > ,MARQUARDSEN                   ,FORSYTHIAVEJ
> > > > > > > > > 38
> > > > > > > > > 1204762526     ,460000699274,MALENE DALGAARD
> > > > > > > > > ,KROGH                         ,CENTRALGÅRDSVEJ
> > > > > > > > > 121
> > > > > > > > > 2103572528 <(210)%20357-2528>     ,460000518039,DORRIT
> > > > > > > > > ,KRISTENSEN                    ,SKJOLDAGERVEJ 92  2
> > > > > > > > > TH
> > > > > > > > > 0507380840     ,460000539850,KARIN
> > > > > > > > > ,SIMONSEN                      ,STRANDGADE 28 N
> > > > > > > > > HOU
> > > > > > > > > 1208520144     ,460000639066,ANNIE
> > > > > > > > > ,BEJART                        ,PRÆSTEGAARDSVEJ
> > > > > > > > > 5
> > > > > > > > > 2403741356 <(240)%20374-1356>     ,460000729054,KAREN
> > > > > > > > > ,EBBESEN                       ,BELLAHOJVEJ
> > > > > > > > > 14
> > > > > > > > > 1903802622     ,460000675467,VINNI MARIE         ,S.
> > > > > > > > > MADSEN                     ,ØSTKÆRVEJ
> > > > > > > > > 27
> > > > > > > > > 2604712478 <(260)%20471-2478>     ,460000701355,METTE
> > > > > > > > > ,OLESEN                        ,BYAGEREN
> > > > > > > > > 25
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >  F3=Exit   F10=Display Hex   F12=Cancel   F15=Services
> > F16=Repeat
> > > > > > > find
> > > > > > > > > F19=Left   F20=Right
> > > > > > > > >
> > > > > > > > > Any idea/help ?
> > > > > > > > > --
> > > > > > > > > This is the Midrange Systems Technical Discussion
> (MIDRANGE-L)
> > > > > > mailing
> > > > > > > > list
> > > > > > > > > To post a message email: MIDRANGE-L@xxxxxxxxxxxx
> > > > > > > > > To subscribe, unsubscribe, or change list options,
> > > > > > > > > visit: http://lists.midrange.com/mailman/listinfo/midrange-l
> > > > > > > > > or email: MIDRANGE-L-request@xxxxxxxxxxxx
> > > > > > > > > Before posting, please take a moment to review the archives
> > > > > > > > > at http://archive.midrange.com/midrange-l.
> > > > > > > > >
> > > > > > > > > Please contact support@xxxxxxxxxxxx for any subscription
> > related
> > > > > > > > > questions.
> > > > > > > > --
> > > > > > > >
> > > > > > > > Luis Rodriguez
> > > > > > > > --
> > > > > > > > This is the Midrange Systems Technical Discussion (MIDRANGE-L)
> > > > mailing
> > > > > > > list
> > > > > > > > To post a message email: MIDRANGE-L@xxxxxxxxxxxx
> > > > > > > > To subscribe, unsubscribe, or change list options,
> > > > > > > > visit: http://lists.midrange.com/mailman/listinfo/midrange-l
> > > > > > > > or email: MIDRANGE-L-request@xxxxxxxxxxxx
> > > > > > > > Before posting, please take a moment to review the archives
> > > > > > > > at http://archive.midrange.com/midrange-l.
> > > > > > > >
> > > > > > > > Please contact support@xxxxxxxxxxxx for any subscription
> related
> > > > > > > > questions.
> > > > > > > --
> > > > > > > This is the Midrange Systems Technical Discussion (MIDRANGE-L)
> > > > mailing
> > > > > > list
> > > > > > > To post a message email: MIDRANGE-L@xxxxxxxxxxxx
> > > > > > > To subscribe, unsubscribe, or change list options,
> > > > > > > visit: http://lists.midrange.com/mailman/listinfo/midrange-l
> > > > > > > or email: MIDRANGE-L-request@xxxxxxxxxxxx
> > > > > > > Before posting, please take a moment to review the archives
> > > > > > > at http://archive.midrange.com/midrange-l.
> > > > > > >
> > > > > > > Please contact support@xxxxxxxxxxxx for any subscription related
> > > > > > > questions.
> > > > > > --
> > > > > >
> > > > > > Luis Rodriguez
> > > > > > --
> > > > > > This is the Midrange Systems Technical Discussion (MIDRANGE-L)
> > mailing
> > > > > > list
> > > > > > To post a message email: MIDRANGE-L@xxxxxxxxxxxx
> > > > > > To subscribe, unsubscribe, or change list options,
> > > > > > visit: http://lists.midrange.com/mailman/listinfo/midrange-l
> > > > > > or email: MIDRANGE-L-request@xxxxxxxxxxxx
> > > > > > Before posting, please take a moment to review the archives
> > > > > > at http://archive.midrange.com/midrange-l.
> > > > > >
> > > > > > Please contact support@xxxxxxxxxxxx for any subscription related
> > > > > > questions.
> > > > --
> > > > This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing
> > > > list
> > > > To post a message email: MIDRANGE-L@xxxxxxxxxxxx
> > > > To subscribe, unsubscribe, or change list options,
> > > > visit: http://lists.midrange.com/mailman/listinfo/midrange-l
> > > > or email: MIDRANGE-L-request@xxxxxxxxxxxx
> > > > Before posting, please take a moment to review the archives
> > > > at http://archive.midrange.com/midrange-l.
> > > >
> > > > Please contact support@xxxxxxxxxxxx for any subscription related
> > > > questions.
> > > >
> > > >
> > > > --
> > > > This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing
> > > > list
> > > > To post a message email: MIDRANGE-L@xxxxxxxxxxxx
> > > > To subscribe, unsubscribe, or change list options,
> > > > visit: http://lists.midrange.com/mailman/listinfo/midrange-l
> > > > or email: MIDRANGE-L-request@xxxxxxxxxxxx
> > > > Before posting, please take a moment to review the archives
> > > > at http://archive.midrange.com/midrange-l.
> > > >
> > > > Please contact support@xxxxxxxxxxxx for any subscription related
> > > > questions.
> > --
> > This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing
> list
> > To post a message email: MIDRANGE-L@xxxxxxxxxxxx
> > To subscribe, unsubscribe, or change list options,
> > visit: http://lists.midrange.com/mailman/listinfo/midrange-l
> > or email: MIDRANGE-L-request@xxxxxxxxxxxx
> > Before posting, please take a moment to review the archives
> > at http://archive.midrange.com/midrange-l.
> >
> > Please contact support@xxxxxxxxxxxx for any subscription related
> > questions.
> --
> This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
> To post a message email: MIDRANGE-L@xxxxxxxxxxxx
> To subscribe, unsubscribe, or change list options,
> visit: http://lists.midrange.com/mailman/listinfo/midrange-l
> or email: MIDRANGE-L-request@xxxxxxxxxxxx
> Before posting, please take a moment to review the archives
> at http://archive.midrange.com/midrange-l.
>
> Please contact support@xxxxxxxxxxxx for any subscription related
> questions.