Preventing Data Mining

[Wayne McAlpine <wayne.mcalpine@xxxxxxxxxxxxxxxxx>]

We make available access to a large database using CGI programs.  More 
and more frequently, I am seeing attempts to download the entire data 
base of millions of records by hitting the server with a script to 
sequentially access every record.

I'm trying to come up with a method of preventing this kind of wholesale 
download.  Many web sites now use a challenge system consisting of an 
authorization word in graphics format that the user must key in before 
being granted access.  The problem is that this solution imposes a 
burden on our legitimate users that shouldn't be necessary, so I'm 
looking for a solution that doesn't involve any additional steps for our 
users.

The first few times this happened, I changed the Apache server 
configuration to deny access to the offending IP address, but it's 
impossible to monitor log files to keep up with the ever-changing IP 
addresses where these attacks originate.

Anyone have any thoughts on how to approach this problem?