RE: Import taking over 24 hours -- RPG400-L

When I wrote the RPG xTools, I tested the Convert to Comma Separated Values
interface by running it over a 46 million record file. The typical item
master size is several hundred million records--the 46 million record file
is a summary file, yikes!
Anyway, using the IBM tools it took about 27 days to convert the file to CSV
format on a module 820 and then a faster (albeit "smaller" processor group)
model 800. 
Using the routine in RPG xTools, it took about 14 days to convert this 46
million record file to CSV format.
After many, many hours of tweaking the parameters, memory allocations and
procedure calls, I got it down to about 7 hours for all 46 million records.

The point here is that you really have to look at the code as it is running
and determine what changes need to be made to optimize the routines. For
example, lots of fields probably means lots of calls to routines that
convert each field. But if you change the routine to call the conversion
once and tell it how many fields to convert, you can reduce the overhead.
This is just one small piece to the larger piece.
What you haven't said is if the conversion code is written in RPG IV or
RPGIII, COBOL, or ILE C.


-Bob Cozzi
www.RPGxTools.com
If everything is under control, you are going too slow.
- Mario Andretti

-----Original Message-----
From: rpg400-l-bounces@xxxxxxxxxxxx [mailto:rpg400-l-bounces@xxxxxxxxxxxx]
On Behalf Of dpalme@xxxxxxxxxxx
Sent: Monday, May 09, 2005 8:44 AM
To: RPG Group
Subject: Import taking over 24 hours

We have been working on an import situation for a couple of months now, and 
I would like to get some feedback from some of you on this.  We have a 
RS/6000 which is running some older cobol programs, we are not in a position

to replace those programs at this time, what we need to do is import that 
data (from standard unix cobol files ACUCOBOL to be exact) into several 
physcial files on our 9406-170.  

We are using MS Access (which I hate but it serves its purpose here) to dump

the data from the cobol files into tab delimited files which are then ftp'd 
over to the 400 and then imported into the physical files.

Everything does work but it is taking around 28 hours for the import process

to complete.

The files can be broken into three groups, customer master data - which 
takes about 20 minutes to complete....the flat file is approximately 8.5mb 
in size.  We have several line item files which are imported into one single

physical file, the flat files for this group are approximately 400mb.  The 
third and final file is a header file which is approx 150mb in size.

We had an outside vendor actually write the code for us which as I said 
before does work, but it is taking a very long time to process.   

If anyone has any ideas or thoughts on how we can speed this process up I 
would love to hear those ideas.  I need to check with the legal department 
before posting the code since I am not sure what contractual obligations we 
are under as far as the code goes.

I do know that he is splitting the fields up by checking for the tab and 
then writing the data to the file. the header file has quite a few fields in

it, and if anyone wants to know the number of fields I can verify that and 
re-post.
-- 
This is the RPG programming on the AS400 / iSeries (RPG400-L) mailing list
To post a message email: RPG400-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/rpg400-l.