There are two kinds of web crawlers, one is goodwill crawler, such as Baidu, Google and other search engines, and the other is malicious crawler, which may use website vulnerabilities to illegally steal website data, or grab website content and occupy server resources. The behavior of malicious reptiles makes us deeply painful, so we must try our best to filter and shield. When websites or some web pages don't want to be crawled by well-meaning crawlers such as Baidu and Googl, we also hope to take some measures.

How to prevent websites from being crawled?

For well-meaning spiders, almost all search engine spiders will abide by the robots protocol. As long as we store an ASCII-encoded text file in the root directory of the website and tell the search engine which pages can't be crawled, the spider of the search engine will follow the agreement and won't crawl the content of the specified page.

But robots are not norms, but a gentleman's agreement, which is only effective for well-meaning reptiles and cannot stop malicious reptiles.

For malicious reptiles, we can take the following measures to identify and intercept them.