Re: Blocking crawlers with robots.txt, but not adsense?

[Ken Sims <mdrg8066@xxxxxxxxxxx>]

Hi David -

On Wed, 04 Mar 2009 13:32:58 -0600, David Gibbs <david@xxxxxxxxxxxx>
wrote:

> User-agent: Mediapartners-Google*

I don't think you want the * on the end.  It's up to each individual
robot as to how it interprets a User-agent other than *, however I
believe it generally is a simple substring check.  The * will cause a
failure on a substring check.

Other than *, think of the User-agent as the first parameter of the
%scan BIF (except that the checking should be case insensitive, but
again that depends on how each individual robot is programmed).

Ken
Opinions expressed are my own and do not necessarily represent the views 
of my employer or anyone in their right mind.