RE: buffering

["Richard Jackson" <richardjackson@xxxxxxxxxxxxxxxxxx>]

Use set object access - it won't take 10 minutes to pin that file in memory,
more like 10 seconds.

I wrote programs to create a test data base.  The records were about 500
bytes.  On a CSC machine, I could write a million records to the file in 16
seconds.  I used very large database blocks.

What language is used to read the data?  If it was RPG, look at the default
blocking.  If you can see the open feedback area, the block size is visible
in there.  I may be in the IO feedback area but I'm not remembering it in
there.  If you used C, use the record IO routines.  I believe that you can
buffer, but not block, byte file transfers.

Use OVRDBF to increase block size.  The optimal block size is 128k because
that is the largest single transfer that you can make to or from an AS/400
disk subsystem.

Store a large block of record data in your program before copying it to the
user space - let MVCL do its magic.

If you preallocate the user space, you are allocating 10 megabytes of space
on disk.  If you are initializing the space, all 10 meg have to be written
before you get to use it.  If you have RAID disk and few disk arms and/or
busy disk arms, it will take longer.

Watch WRKSYSSTS for database faulting.  On the AS/400, database doesn't
fault unless its pages are stolen.  Faulting means that pressure for memory
is high.  Enlarge the pool or move the job to a less-busy pool.  I assume
that the job has no trouble getting to an activity level and that it is
running at a high priority.

You didn't say if it was slow at the beginning, in the middle, or at the end
...

I think that this transfer should take less than 20 seconds on a smallish
RISC machine.  I would be surprised if you could get it much under 4 seconds
on any RISC machine available today - unless the input file was already in
memory.

Are you using Dick's new sort APIs?

Richard Jackson
mailto:richardjackson@richardjackson.net
http://www.richardjacksonltd.com
Voice: 1 (303) 808-8058
Fax:   1 (303) 663-4325

-|-----Original Message-----
-|From: owner-mi400@midrange.com [mailto:owner-mi400@midrange.com]On
-|Behalf Of Leif Svalgaard
-|Sent: Saturday, October 07, 2000 1:57 PM
-|To: MI400 List
-|Subject: buffering
-|
-|
-|As part of my ongoing investigation of sorting large files,
-|I'm reading the files myself into storage to sort them there.
-|I have a question about buffering. It doesn't make sense
-|that it takes 15 minutes to read 10Mb (1,000,000 records
-|of 10 bytes each), so I must not have the right buffering
-|set up. What controls that? I was under the (mistaken ?)
-|impression that suitable buffering was done by OS/400
-|so I don't have to worry about it. If that's the case, it
-|is doing a poor job. More likely, I need to do something
-|to get better buffering. What?
-|
-|Leif Svalgaard
-|
-|+---
-|| This is the MI Programmers Mailing List!
-|| To submit a new message, send your mail to MI400@midrange.com.
-|| To subscribe to this list send email to MI400-SUB@midrange.com.
-|| To unsubscribe from this list send email to MI400-UNSUB@midrange.com.
-|| Questions should be directed to the list owner/operator:
-|dr2@cssas400.com
-|+---

+---
| This is the MI Programmers Mailing List!
| To submit a new message, send your mail to MI400@midrange.com.
| To subscribe to this list send email to MI400-SUB@midrange.com.
| To unsubscribe from this list send email to MI400-UNSUB@midrange.com.
| Questions should be directed to the list owner/operator: dr2@cssas400.com
+---