bot invasion
Posted: October 18th, 2018, 6:10 am
Recently I had the same problem with our typo3 website at work. For the htaccess file there is a fast way to forbid folders to those bots
User-agent: *
Disallow: /subdir1/
Disallow: /subdir2/
Disallow: /subdir3/
I think this works pretty good, so they better crawl the page content not the file folders
User-agent: *
Disallow: /subdir1/
Disallow: /subdir2/
Disallow: /subdir3/
I think this works pretty good, so they better crawl the page content not the file folders