Re: Reading a big file

["Ari Samson" <ari@xxxxxxxxxxxxxxx>]

boothm@earth.goddard.edu wrote

> The file I am concerned with has 5 million records.  I want to go through
> this file making selections based upon choices made for several fields. 

During one of the days before I lost all my hair I was at an account who
seemed to love the idea of big mainframe flatfiles on the 400. It drove me
batty. At one time they had two very interesting files. One had 25 million
records each one about a k (1000 bytes) in length, and the other one had 45
million records each one about 90 bytes in length. We had to do some very
strange things with these files. 

One important lesson to remember, from testing we seemed to discover that
after a file was larger than 10 million records any extrapolation of data
base update could get thrown out the window. It was very hard to estimate
anything. People who have more knowledge of how the index building works on
the 400 can comment here. 

Anyway, for the small length record, we discovered that building an access
path on the fly either through OPNQRYF or literally throwing together some
DDS with select omit criteria ONLY worked great. This especially worked
well when I needed to pass parms to the process. The trick was to do the
select omit to grab your records. Toss them to a physical file and then
sort using FMTDTA or something similar.  Most 400 tools unfortunately sort
before they select. It can drive you quite mad if you only want a small
number from a large file. 

For the large length record, it didn't seem to matter what we did we were
basically SOL. Anyway, I speny months doing RTI of processes built over
these files. If you really want some hard numbers I can probably dig them
out of whatever pile they're presently buried in and let you know. Just let
me know.

Ari Samson  
Tech Software
(516) 627-3800
email: ari@techsoftinc.com

"If WWI was the war to end all wars and WWII was the war to end that theory
then does that mean that WWIII will be the war to end all theories?"

* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
* This is the Midrange System Mailing List!  To submit a new message,   *
* send your mail to "MIDRANGE-L@midrange.com".  To unsubscribe from     *
* this list send email to MAJORDOMO@midrange.com and specify            *
* 'unsubscribe MIDRANGE-L' in the body of your message.  Questions      *
* should be directed to the list owner / operator: david@midrange.com   *
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *