Re: FTP from unix/linux system causing EOL issues

[Don Brown <DBrown@xxxxxxxxxx>]

Hi Scott,

Caught by the assumed :-)

Yes we are ftp'ing directly to a PF

Yes we are using the IBM provided ftp client.

Yes we are explicitly using ascii mode.

If I ftp from a windows machine the CRLF are removed

If I ftp from the linux system the CRLF is replace with a LF (0D)

Scott wrote: If you are transferring directly to a PF, the IBM FTP client 
should 
already know that in ASCII mode the records will end in CRLF, and remove 
that for you. 

I wish that was the case but that is not what is happening - see below 
data from DSPPFM + F10 + F11

From the trail

> > After ftp'ing the file directly to a physical file we have a end of 
line
> > character of hex 0D
> >
> > Here is an example
> > ,,,13/02/23,12:50,$20.55,$0.00,006793,V31142█
> > 666FF6FF6FF6FF7FF65FF4FF65F4FF6FFFFFF6EFFFFF0444
> > BBB13102123B12A50BB20B55BB0B00B006793B531142D000


> > With files that we ftp from a windows server there is no end of line
> > character at all as per the following example
> >
> > Pay Period,,2302,,Account Number ,,4770,,,,,,,,,,
> > D8A4D8989866FFFF66C889A9A4DA9889466FFFF666666666644444
> > 7180759964BB2302BB133645305442590BB4770BBBBBBBBBB00000

Thank you for you  responses

Don





From:   "Scott Klement" <sk@xxxxxxxxxxxxxxxx>
To:     midrange-l@xxxxxxxxxxxxxxxxxx
Date:   16/02/2023 08:39 AM
Subject:        Re: FTP from unix/linux system causing EOL issues
Sent by:        "MIDRANGE-L" <midrange-l-bounces@xxxxxxxxxxxxxxxxxx>



Hi Don,

I guess I assumed you were FTP into an IFS stream file.  Are you loading 
it into a physical file?

If you are transferring directly to a PF, the IBM FTP client should 
already know that in ASCII mode the records will end in CRLF, and remove 
that for you.  If you are using a different FTP client than the one that 
came with the OS, you should check with whomever is responsible for 
supporting that client.

If you are FTPing to the IFS, then you can simply set the CCSID of the 
object to whichever type of ASCII it's encoded with.

-SK

On 2/15/23 3:58 PM, Don Brown via MIDRANGE-L wrote:
> Hi Scott,
>
> Thanks for your input ...
>
> The file being retrieved from the linux ftp server is in ascii.
>
> If I use ascii mode on the transfer the file is converted to ebcidic.
>
> If I use bin(ary) mode it is not converted. (I know as this was my first
> thought and I tried it)
>
> Now I am presuming ccsid's come into this somehow to convert from ascii 
to
> ebcidic but I do not know how.
>
> Or do I need to ftp the file to the IFS and then copy it specifying the
> from ccsid ?
>
> Our current solution of checking and replacing the EOL character(s) is
> working but is realy a brute force and ignorance solution.
>
> As we are likely to encounter more unix/linux ftp servers in the future
> having a more specific approach is preferable.
>
> And I presume we will have exactly the same issues using sftp/scp
> etcetera.
>
> I am sure others have had this issue or will have so hopefully there is 
a
> simple process providing the solution..
>
> Thanks
>
> Don
>
>
>
>
>
> From:   "Scott Klement" <midrange-l@xxxxxxxxxxxxxxxx>
> To:     midrange-l@xxxxxxxxxxxxxxxxxx
> Date:   16/02/2023 06:45 AM
> Subject:        Re: FTP from unix/linux system causing EOL issues
> Sent by:        "MIDRANGE-L" <midrange-l-bounces@xxxxxxxxxxxxxxxxxx>
>
>
>
> Hi Don,
>
> Indeed, that is the way ASCII mode is supposed to work.  in ASCII mode,
> the sending computer converts the end-of-lines to CRLF (hex 0d0a).  The
> receiving computer should, then, convert these to it's own native 
format.
> For example, if you are on a Windows PC and connecting to Unix, when you
> switch to ASCII mode, it will send CRLF, and since it's the norm on
> Windows, it'd keep it as-is when saving the file.
>
> On the other hand, if you are on a Unix machine, the receiving FTP
> client would convert the CRLF to just LF.  Because that's the norm on
> Unix.
>
> This is the purpose of ASCII mode.
>
> Since IBM i does not really have a standard between LF and CRLF, it will
> just save the CRLF.
>
> Maybe you should be receiving the data using binary mode so that it's
> not converted?
>
> -SK
>
>
> On 2/15/23 1:30 AM, Don Brown via MIDRANGE-L wrote:
> > We are transferring files from a new ftp server hosted on a unix/linux
> > system.
> >
> > The transfer is using ascii mode
> >
> > If you look at the ascii file there are CRLF  (0A0D) at the end of each
> > line
> >
> > After ftp'ing the file directly to a physical file we have a end of 
line
> > character of hex 0D
> >
> > Here is an example
> > ,,,13/02/23,12:50,$20.55,$0.00,006793,V31142█
> > 666FF6FF6FF6FF7FF65FF4FF65F4FF6FFFFFF6EFFFFF0444
> > BBB13102123B12A50BB20B55BB0B00B006793B531142D000
> >
> > Our process to then read the file wich is in csv format was then 
failing
> > on the end of line character.
> >
> > We have added some code to replace the hex 0D with a blank and this has
> > enabled the file to be processed but I have not been able to find any
> > other option.
> >
> > So just asking the group if someone has hit this issue before and how
> they
> > resolved the problem
> >
> > With files that we ftp from a windows server there is no end of line
> > character at all as per the following example
> >
> > Pay Period,,2302,,Account Number ,,4770,,,,,,,,,,
> > D8A4D8989866FFFF66C889A9A4DA9889466FFFF666666666644444
> > 7180759964BB2302BB133645305442590BB4770BBBBBBBBBB00000
> >
> >
> > Any suggestions welcomed
> >
> > Don
> >
> >
> >
> > --
> > This email has been scanned for computer viruses. Although MSD has 
taken
> reasonable precautions to ensure no viruses are present in this email, 
MSD
> cannot accept responsibility for any loss or damage arising from the use
> of this email or attachments.