Re: Unexpected division quotient

OK. Stupid alternative, but it might help build consensus and or broaden knowledge:

Why not
a for loop, up to ds_Inp,RevDec
( a * .01 )
and then C = a

:)


On 7/2/2012 4:26 PM, Anderson, Kurt wrote:

I should have explained why I thought the two lines I gave should always work, so I can be corrected if I'm wrong.

This should work b/c even though it's floating point, there is no division occurring. Only division causes a lack of decimal precision with floating point?
C = a * (10 ** (ds_Inp.RevDec * -1));

This should work because the %int bif will override the resulting floating point value to be an integer which in turns causes the division to occur as expected?
C = a / %int(10 ** ds_Inp.RevDec);

Thanks for the feedback,
Kurt

-----Original Message-----
From: rpg400-l-bounces@xxxxxxxxxxxx [mailto:rpg400-l-bounces@xxxxxxxxxxxx] On Behalf Of Anderson, Kurt
Sent: Monday, July 02, 2012 4:21 PM
To: RPG programming on the IBM i / System i
Subject: RE: Unexpected division quotient

Thanks Chuck, I wouldn't have thought of checking the Visual Age RPG documentation (j/k).

Thanks for pointing that out, though.

For my own sanity, based on that, can I trust that both of the following will always work?
C = a * (10 ** (ds_Inp.RevDec * -1));
C = a / %int(10 ** ds_Inp.RevDec);

ds_Inp.RevDec is 4,0 zoned (which is why I used %int instead of %dec).

-Kurt

So is it simply luck that using the exponential operator on a negative value is giving me the value I want?

-----Original Message-----
From: rpg400-l-bounces@xxxxxxxxxxxx [mailto:rpg400-l-bounces@xxxxxxxxxxxx] On Behalf Of CRPence
Sent: Monday, July 02, 2012 4:07 PM
To: rpg400-l@xxxxxxxxxxxx
Subject: Re: Unexpected division quotient

Searching the v5r4 InfoCenter on the following tokens, yields the quoted text that follows:
exponentiation rpg

"VisualAge RPG Language Reference

Precision of Intermediate Results

Table 52 describes the default precision rules in more detail.

Table 52. Precision of Intermediate Results
Operation Result Precision

Note:
The following operations produce a numeric result. Ln is the length of the operand in digits where n is either r for result or a numeric representing the operand. Dn is the number of digits to the right of the decimal point where n is either r for result or a numeric representing the operand. T is the temporary value.

Note that if any operand has a floating point representation (for example, it is *the result of the exponentiation operator* ), the result also is a floating point value and the precision rules no longer apply.
A floating point value has the precision available from double precision floating point representation.

...
"

Regards, Chuck

On 02 Jul 2012 15:45, Anderson, Kurt wrote:

IBM i 7.1

There is a situation where we have an integer sent to us carrying an
amount value. Another field contains the number of implied decimals in
that integer.

Here is what I'm seeing:

dividend = 418
divisor = 10 to the power of 4 (implied decimal value) = 10000.
quotient = .041799 (but I'm expecting .041800)

C = a / (10 ** ds_Inp.RevDec);
// C = .041799


I have tried with and without the H-spec ExPrOpts( *ResDecPos ).

When I break down the code into two lines, it does work. And 2 lines
is fine, b/c this is still better than the 19 lines of code we used to
have, but I was would like to understand why this is happening.

Example of how to get .041800.
B = (10 ** ds_Inp.RevDec);
C = a / b;
// C = .041800

--
This is the RPG programming on the IBM i / System i (RPG400-L) mailing list To post a message email: RPG400-L@xxxxxxxxxxxx To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives at http://archive.midrange.com/rpg400-l.

--
This is the RPG programming on the IBM i / System i (RPG400-L) mailing list To post a message email: RPG400-L@xxxxxxxxxxxx To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives at http://archive.midrange.com/rpg400-l.