RE: 2. Re: How to compress data in RPGLE and save it in a file and -- RPG400-L

After reading this, you have to wonder what compressing/decompressing the 
data's going to do to CPU utilization (especially when you start decompressing 
interactively). Compress and decompress operations are usually CPU intensive so 
if you start running out of interactive CPW, it's going to cost big bucks to 
upgrade it.

If you're looking for disk space savings, it seems to me the better approach 
may be to normalize the database (I couldn't imagine that what you have now is 
normalized).

Have you thought about variable length database fields? This could be a pretty 
easy way to free up disk space.

What about archiving the data to tape? This is an old solution to the out of 
disk space problem but maybe it's worth investigating.

If compressing is the only way to go, you may want to take a look outside of 
RPG land to see what's available. I bet you can find plenty of C routines (like 
maybe the zlib algorithm -- you can get a version for OS/400 at 
http://homepage1.nifty.com/uzaemon/#download) to do this and I'm sure there's 
Java as well. You could probably even port these to RPG if you really wanted. 
One thing to watch out for is the compression algorithm. I don't think the 
patent has expired on LZH compression yet (but it is coming to an end pretty 
soon if it hasn't already) and I'm sure there are still tons of examples using 
this algorithm and others that require licensing.

Of course, you could always take the approach we have and just buy more disk 
(it's cheaper by the skid full!). You will see some benefit in throughput when 
you do this since you'll have more disk arms.

Matt

-----Original Message-----
From: Tanveer, Mohammad [mailto:mtanveer@xxxxxxxxxxxxxxxxxx]
Sent: Wednesday, December 31, 2003 12:37 PM
To: 'rpg400-l@xxxxxxxxxxxx'
Subject: 2. Re: How to compress data in RPGLE and save it in a file and


Hi Scott,

Sorry for not explaining what I wanted.

We are expecting around one million orders a day, now some how on daily
basis we need to archive this data and be able to use it as needed.  

Our thinking is that we will store the lookup data in regular file layout
(index file), which will be used to search records but rest of the data will
be in compress format or files.  User can search the records using Index
file and then we can uncompress the required data and use it.  

I can easily do it on PC platform but wondering if there is any utility or
functionality within OS/400 which can help us to do what we are trying to
do.  Meaning save the disk space by compressing it and also be able to
recover data from compress mode within seconds.

Any idea how to solve this issue?  It's all regular data (numbers/strings).
For one order we have 100s of files and there are 100s of fields in each one
of them.  Don't ask why we have so many files and fields.  (FYI: It's all
custom development of many years). We would like to compress this data by
order number so that we can easily retrieve it.
<snip>