robots.txt wildcard question

[markzz]

Tweet

I need all search engines to exclude any files that end in .axd on my website.


User-agent: Mediapartners-Google*
Disallow: *.axd

User-agent: *
Disallow: *.axd

User-agent: Googlebot*
Disallow: *.axd


Will this work? I have heard that the wildcard only works with Google?

[Brendon]

All you need is this section :

User-agent: *
Disallow: *.axd

Any "good" bot should see that, and obey it. Bad bots you'll have to deal with yourself (exclusion by IP and / or UA)

Note that browsers are never affected, so there will be no user impact.

>> I have heard that the wildcard only works with Google?

All bots that obey robots.txt should be fine with wildcards. Try and make sure you use Unix line endings though, some bots need it to parse the file correctly. If you don't know how, get a free text editor that supports them (I like NoteTab, personally), and create your file as normal, just save it using the "Unix line endings" option

[markzz]

I don't think that is correct - the wildcard (*) is [i]not[/n] standard robots.txt protocol.

This is from Matt Cutt's Blog:

"Finally, each search engine has slightly different extra options that they support. For example, Google permits wildcards (*) and the “Allow:” directive MSN and Yahoo do not support wildcards - it's not in the official robots.txt protocol"

Can anyone help me with this, or do I need to exclude all the files manually?

[Brendon]

Well, if you're going to get technical, robots.txt isn't a standard at all...

User-agent: * will work fine though; wildcards are permitted in the UA field. You will need to move all your .axd files to a specific folder, and simply disallow that folder, so :

User-agent: *
Disallow: /axd

where /axd contains all your .axd files would do just fine, and will be compliant. Otherwise you could simply list every specific .axd file thus :

User-agent: *
Disallow: foo.axd
Disallow: bar.axd
Disallow: snafu.axd
etc...

[markzz]

unfortunately that is not possible. they are virtual files created by the site... guess i'll have to exclude them manually...

robots.txt isn't a standard? well we'll call it a pseudo-standard as all major SEs use it lol...

[Brendon]

>> they are virtual files created by the site

Couldn't you jig the site to assign a URL to the files of <root>/axd/filename.axd? Use a JS document.write statement to do the same? Or just us JS links to the files anyway, so SEs won't "see" the links in the first place?

>> robots.txt isn't a standard? well we'll call it a pseudo-standard as all major SEs use it lol...

Technically, no it isn't a standard. It is similar to the more recent "nofollow" attribute in that the major SEs have agreed to use / support it, as it helps solve a problem for everybody. Consider how many bots you see who plainly DON'T respect robots.txt, there's frickin'thousands of 'em.... so, no, it's not a standard