Re: Buffered reading of stream files

[Adam Glauser <adamglauser@xxxxxxxxx>]

Christen, Duane J. wrote:
> I use the posix(unix) open(), close(), read()... APIs to process stream file, similar to the C functions.

That's what I've used up to now too, thanks to Scott Klement's examples 
and IFSIO_H copybook.

I have a couple of reasons for considering the C/C++ runtime functions. 
 As Paul said, it's much more widely tested.  Also, as far as I can 
tell, I don't have to worry about maintaining different buffers if the 
routine is called for multiple files in the same actgrp.  Most of the 
time my records will be newline delimited, so I can just use fgets() and 
be done with it.  What are the reasons you prefer the POSIX functions?

> In general I allocate 4M of memory for the file buffer and read the file in 4M blocks, and process the buffer. The tricky part is dealing with "records" that cross the 4M boundary, part is in memory part is still in the file.

Yeah.  I think I've managed to deal with this in a reasonable way.  The 
other tricky thing was that the record delimiters may fall partially 
into one buffer and partially into another.  FWIW, I just reload the 
buffer when I get less than a delimiter's length away from the end of 
the buffer.

Why 4M?

> The big problem I see is performance, just like reading one record at a time from a DB could be a performance hit, reading one byte at a time from a stream file is almost guaranteed to be a poor performer.

Right, as I stated earlier, I'm already doing some buffering.