robots.txt is a txt file desgined to keep robots out of stuff you don't want them to see.
we use one as well... https://hondaswap.com/robots.txt
ia_archiver is the user agent name for the bot that works for archive.org. This lets them save everything.
slurp crawl delay because yahoo bot goes nuts (like 300k requests a second some times)
everyone else, i don't want to them to index any junk content. all of our pages that i care about indexing in search engines end in / or .html. Thus, any php pages, js, cfm, asp, etc etc i do not want the search bots to waste any time indexing, so i dsiallow them.
now, obviously there's not user-agent of "kids".... so google pulled a funny and is allowing treats but not tricks
That or my head would explode on it's own. I kick ass at basic html and css and flash but if anyone tells me they want php or to use a database I am fucked.
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.