A Performance Question (Caching)

["Chris Pando" <chris.pando@xxxxxxxxx>]

Performance Questions:

We are resequencing our employee numbers. This requires me to read all
the records in a file and, for every field containing employee number,
see if the number is to be converted. If so, the field is changed, and
the record updated.


Initially, I chained to the conversion file (containing the before and
after employee numbers).

This takes a long time to run (especially over a General Ledger
detail with 100 000 000 records), and we have a limited amount of
downtime in which to run the conversion, so we started looking
for areas that could be optimized.

As a first step, rather than chaining to the conversion file, I
loaded the entire thing into the heap, and used a binary search
to find my conversion values (see note 2):


The version using binary search and the heap runs in *half* the
time of the version using the chain (verified using Performance
Explorer).

My question is why? My conversion file is less than 130K, and we've
got roughly 5G memory - why doesn't the operating system cache the
conversion file, rather than forcing me to do so explicitly?


TIA,

Chris Pando


Notes: (best viewed w/ fixed font)

(note 1)

 Read input bufferDS;
 DoW (Not %Eof(input));

    change = *Off;

    For $I = 1 to fldCnt;
       QDAID = %SubSt( bufferDS : fldOff($I) : 8 );
       Chain QDAID fy81503;
       If ( %Found(fy81503) );
          %SubSt( bufferDS : fldOff($I) : 8 ) = QDACID;
          Change = *On;
       EndIf;
    EndFor;

    If ( change );
       Update input bufferDS;
    EndIf;

    Read input bufferDS;
 EndDo;

Rather than use an externally defined file, I read the file
into a an array named buffer (overlaying the data structure
bufferDS).

I have an array named fldOff containing the offset within the
buffer for each field that can contain employee number. The
file FY81503 contains two fields:

QDAID  Employee Number, before
QDACID Employee Number, after

(note 2)

Read input bufferDS;
DoW (Not %Eof(input));

   change = *Off;

   For $I = 1 to fldCnt;
      $$@ = bsearch( %Addr(buffer(fldOff($I)))
                   : heap@
                   : frmCnt
                   : 16
                   : %PAddr('BSEARCHCOMP')
                   );
      If ( $$@ <> *NULL );
         change = *On;
         %SubSt( bufferDS : fldOff($I) : 8 ) = $$ACID;
      EndIf;
   EndFor;

   If ( change );
      Update input bufferDS;
   EndIf;

   Read input bufferDS;
EndDo;

$$@ is defined as:


d                 ds                  Based($$@)
d $$AID                   1      8
d $$ACID                  9     16


bsearchcomp() is defined as:

p bSearchComp     b
d                 pi            10i 0
d a@                              *   Value
d b@                              *   Value

 /free
  return memcmp( a@
               : b@
               : 8
               );
 /end-free
p bSearchComp     e

I loaded the heap like this:

heap@ = %Alloc(HEAPALLOCSIZE);
$$@ = heap@;
Read fy81503;
DoW ( Not %Eof(fy81503) );
   frmCnt = frmCnt + 1;
   $$AID  = QDAID;
   $$ACID = QDACID;
   $$@    = $$@ + 16;
   Read fy81503;
EndDo;