Re: Number Precision ?

[boldt@xxxxxxxxxx]

Shen wrote:
>0031.01 DMARKUP1          S              5  2
>0031.02 DMARKUP2          S              5  2
>0031.03 DLIST                   S             11  2
>0031.04 DCOST                 S             11  2
>0031.05   **  LIST = 42.49
>0031.06   **  COST = 14.54
>0031.07 C                   EVAL      MARKUP1 = (LIST-COST)/(COST/100)
>0031.08   **  MARKUP1 = 192.00
>0031.09 C                   EVAL      MARKUP2 = (LIST-COST)/(COST/100.00)
>0031.10   **  MARKUP2 = 192.20
>
>Why MARKUP1 = 192.00 and MARKUP2 = 192.20 ? That's the question
>about the number precision.

In some other fora, you'd simply get the answer: RTFM.
(Read The (ahem) Friendly Manual).

In this case, the appropriate reference is chapter 21 of
the ILE RPG Reference manual.  More specifically, look at
table 35 (on page 406 of the V4R2 hardcopy book).

Let's look at the first expression:  Each of the intermediate
results must be saved in a temporary variable with a specific
precision.  Table 35 describes how the precisions are determined.
Here's how the arithmetic is performed:

1)  (LIST-COST)   (11,2)-(11,2)  --> result has precision (12,2)
2)  (COST/100)    (11,2)/(3,0)   --> precision (30,21)
3)  (LIST-COST)/(COST/100)   (12,2)/(30,21) --> precision (30,0)

As you can see, the decimal positions are lost!  This is a very
common situation when coding several division operations in one
expression.

So, what do you do?  Here are some alternatives:

1) Code the expression in 2 separate statements.
2) Force the result of (COST/100) to a specific precision
   using built-in function %DEC.
3) Use op-code extender R, or H-Spec keyword EXPROPTS(*RESDECPOS)
   (as documented in Chapter 21).  This forces the decimal positions
   of the intermediate results of an expression to never drop below
   the decimal positions of the assignment target.

Cheers!  Hans

Hans Boldt, ILE RPG Development, IBM Toronto Lab, boldt@ca.ibm.com


+---
| This is the Midrange System Mailing List!
| To submit a new message, send your mail to MIDRANGE-L@midrange.com.
| To subscribe to this list send email to MIDRANGE-L-SUB@midrange.com.
| To unsubscribe from this list send email to MIDRANGE-L-UNSUB@midrange.com.
| Questions should be directed to the list owner/operator: david@midrange.com
+---