Re: Encryption Problem.

[Eduard Sluis <eduard@xxxxxxxxxxxxxxxxxxxxxxx>]

Hi all,


Thanks to private help from the group we are now able to decrypt successfully.
The problem was that we took everywhere the EBCDIC/ASCII differences in account 
but not in regard to the key....
After taking care of decrypting with the correct key the result started to 
become readable.

Kind regards,
Eduard Sluis.

________________________________
From: Eduard Sluis <eduard@xxxxxxxxxxxxxxxxxxxxxxx>
To: Midrange Systems Technical iSeries Discussion <midrange-l@xxxxxxxxxxxx>
Sent: Tue, December 7, 2010 8:15:05 PM
Subject: Encryption Problem.

Hi all,

We have problems to get a correct result when Decrypting on the IBM i with RC4.
What is the case?

We are communicating with an external device (not an IBM i).
That device Encrypt with RC4.
The results is made Base64 to get a string that can be communicated.
That string is send to the IBM i.
We decode on the IBM i and get the same Hex code as the device had before the 
Base 64 step.
We decrypt the RC4 and make the result EBCDIC on the IBM i but we do not get the 

original input back.

So the steps we take are:
On the device:
Have the Clear text.
Encrypt with RC4.
Encode with Base64.
Send to the IBM i.
On the IBM i:
Decode the Base64.
Decrypt the RC4.
Translate to EBCDIC.

Unluckily after the decryption on the IBM i we do not get the correct result.
We checked and tried a lot of different things but we could not come to a 
resolution.
We also tried the two different decryption functions for RC4 we have found.- 
Qc3DecryptData
- Cipher( p_ENCOPN:RC4_Controls:p_ENCCLS);To our surprise they gave a different 
results that both are not the original input.
Sure we are doing something wrong but we cannot figure out what and resolve it 
as we normally can and do.

I hope some of you has more experience and can point out what we are doing 
wrong.
The code snippets of the Decryption we have tried are:

 key_ctx = *ALLx'00';                                           
 %subst(key_ctx.stream:1:%len(key)) = key;                      
 key_ctx.len = %len(key);                                       
                                                                
RC4_Controls = *ALLx'00';                                       
RC4_Controls.funct_id = x'0013';                                
RC4_Controls.datalen  = %size(data);                            
RC4_Controls.operation = x'00';         // 0=Encrypt,1=Decrypt  
RC4_Controls.p_key_ctx = %addr(key_ctx);                        
                                                                
p_recv = %addr(encrypted);                                      
p_src  = %addr(data);                                           
                                                                
cipher( p_recv: RC4_Controls: p_src);                           

And:

Qc3DecryptData(encdata:nRTNlen:MyAlgo:'ALGD0300':myKey:'KEYD0200':
                          
ANY_CRYPTO_SRV:CRYPYO_SRV:szData:%size(szDate):nRtnLen:apiError);

If we follow the sequence of the steps we have:
- Clear text on the device: - Hex Input text:
This iS a Test!
54 68 69 73 20 69 53 20 61 20 54 65 73 74 21- Clear text encrypted with key: 
12345678901234567890

- Hex Encrypted text:9E 44 9C 72 78 33 41 61 A5 A9 4D 0C 7B C5 FADrx3Aa¥©M{Åú
- Base 64 Encoded:
nkSccngzQWGlqU0Me8X6
- Hex Base 64 Encoded:95 92 E2 83 83 95 87 A9 D8 E6 C7 93 98 E4 F0 D4 85 F8 E7 
F6
 
- Base 64 Encoded received on IBM i:
nkSccngzQWGlqU0Me8X6
- Hex as recieved on IBM i (see is same as sent):95 92 E2 83 83 95 87 A9 D8 E6 
C7 93 98 E4 F0 D4 85 F8 E7 F6
- Result of Base 64 decoding
ÆàæÊÌ /vz(#E³
- Hex of Base 64 Decoding (see is same as Hex input to Base 64 Encoding on 
device)!9E 44 9C 72 78 33 41 61 A5 A9 4D 0C 7B C5 FA
- Clear text result of Xßü;QX¹Õ#X is hex:
E7 59 DC 5E 2D D8 2E 2D E7 DA EF FF 7B E7 24- Clear text result 
of Cypher:Qc3DecryptData:
2{k6âÈü¥n^äx is hex: 
F2 0A C0 92 F6 42 74 DC 29 B2 95 B0 2F 43 A7
 
Both differ from each other!! and both differ dearly from the the expected: 
9E 44 9C 72 78 33 41 61 A5 A9 4D 0C 7B C5 FA
 
Clearly we have not tried the way back yet....
When we do all steps on the IBM i we do get the correct results (with both 
methods but not intermingled).
We would like to assume that RC4 is RC4 and equal input would create equal 
output.
Even with the two IBM i methods that seems not the case.....
 
If we use different tools on the Internet to Encrypt (with one tool) and Decrypt 

(with another tool) we get the same results.
We do not believe that the problem is created on the Device.
Even if that would be the case we would need to get the same wrong result with 
the two decryption methods.  
 
Who will be able to help us??
 
Kind regards,
Eduard Sluis.