A: uspider will automatically adjust the access density according to the load capacity of the server. After a period of continuous access, the spider will pause for a while to prevent increasing the access pressure on the server. So in general, the server of your website will not cause too much pressure.
2. Why does spider keep crawling my site?
A: The spider will keep crawling the newly generated or continuously updated pages on your website. In addition, you can also check the site access logs to see if the spider's visits are normal, to prevent someone from maliciously impersonating the spider to crawl your site frequently.
3. I don't want my site to be visited by spider, what should I do?
A: The spider abides by the Internet robots protocol. You can utilize the robots.txt file to prohibit the spider from accessing your site altogether, or to prohibit the spider from accessing some of the files on your site.
4. Why is my site still searchable in Baidu even though I've added robots.txt?
A: Because it takes time to update the search engine index database. Although the spider has stopped accessing the pages on your site, it may take two to four weeks for the Baidu search engine database to be cleared of the web indexing information that has been created. Also check if your robots configuration is correct.
5. I want my site's content to be indexed by Baidu but not saved as a snapshot, how do I do that?
A: The spider adheres to the internet meta robots protocol. You can use web page meta settings to make Baidu display only the indexing of the web page, but not show a snapshot of the page in the search results.
As with robots, it takes time to update the search engine index database, so although you have disabled Baidu from displaying a snapshot of the page in search results through meta on the web page, it may take two to four weeks for the Baidu search engine database to take effect online if it has already indexed the page.
6. How long before Baiduspider re-crawls my pages?
A: The Baidu search engine is updated weekly, and web pages have different update rates depending on their importance, with a frequency of between a few days and a month, and Baiduspider will revisit and update a web page.
7. Baiduspider crawl caused by bandwidth congestion?
A: Baiduspider's normal crawl does not cause bandwidth congestion on your site, which may be due to malicious crawling by someone posing as a baidu spider. If you find an agent named Baiduspider crawling and causing bandwidth blockage, please contact us as soon as possible. You can also send us feedback to the Baidu Web Complaint Center, or provide us with the access logs of your site at that time, which will be more conducive to our analysis.