What Is Baiduspider (Baidu Spider) and How to Manage It

Baiduspider, also written Baidu spider, is the web crawler of Baidu, China’s largest search engine. It visits web pages and adds them to Baidu’s index, and it identifies itself with user agents that contain “Baiduspider”, such as Baiduspider-image or Baiduspider-render. You can verify its visits with a reverse DNS lookup and control it with robots.txt.

Key takeaways

• Baiduspider is the crawler that builds Baidu’s index. Its desktop and mobile user agents both contain “Baiduspider/2.0”.
• Baidu uses one agent per product, such as Baiduspider-image, Baiduspider-video or Baiduspider-news, and it has used Baiduspider-render to load CSS and JavaScript since 2017.
• To check that a visit really comes from Baidu, run a reverse DNS lookup: the hostname must end in .baidu.com or .baidu.jp, and resolve back to the same IP address.
• The crawler follows robots.txt, but Baiduspider-cpro and Baiduspider-ads, which work for Baidu’s advertising customers, do not.
• If you target China, let the crawler in: host your site for fast access from China and keep important content in the HTML.

What Is Baiduspider?

Baiduspider is the official name of the crawler used by Baidu, the most important search engine in China. In its FAQ about Baiduspider, Baidu describes it as the program that visits pages on the internet and builds their information into the Baidu index, so that users can find your site when they search.

Search bots, or spiders, regularly visit your pages, and what they find decides how your content can rank. Every search engine sends its spiders automatically, but you keep some control over how they interact with your website. When you run a website in China, or in Chinese, you will deal with Baiduspider. It works like Googlebot, with differences worth knowing.

Google search is blocked in mainland China, so Baiduspider is the crawler that matters when you target the Chinese market. Making a website easy for it to crawl is part of the work of our Baidu SEO agency.

How Does Baiduspider Work?

As with any other search engine crawler, Baiduspider is an automated program that regularly checks the content of your website to collect the information used to index your pages.

Every time Baiduspider visits your pages, it looks at their structure, the quality of their content, their updates, their keywords and more. The crawling process has two steps: the spider fetches the page and stores it, then it lists the links on the page to check them later.

With the data collected, Baidu ranks your content. A page can reach the top of the search results if Baidu finds it valuable, or be downgraded if it does not meet Baidu’s guidelines. Baidu adjusts how often Baiduspider visits a site according to the capacity of the server, the quality of the site and how often it is updated.

You probably neither want nor need Baiduspider to visit all your pages. You need to understand both how to block Baiduspider and how to make it like your content.

Does Baiduspider Crawl Websites Outside China?

Many webmasters wonder if Baiduspider crawls Chinese websites only. It does not: the crawler works like any other visitor, so if a user in China can access your website, Baidu can crawl it too. But a website hosted overseas that loads slowly from China will rank poorly, and in 2017 Baidu warned that Chinese websites hosted on servers abroad would have display problems in its results. If your target is the Chinese market, have your website hosted in China.

Baiduspider User Agents

To collect information, Baidu uses a different user agent for each of its products. This list follows Baidu’s FAQ, with the rendering agent Baidu added in 2017:

Baidu’s Product Name	Baidu’s User Agent
Baidu Web and Mobile Search	Baiduspider
Baidu Image Search	Baiduspider-image
Baidu Video Search	Baiduspider-video
Baidu News Search	Baiduspider-news
Baidu Bookmark Search	Baiduspider-favo
Baidu Business Search	Baiduspider-ads
Baidu Union	Baiduspider-cpro
Page rendering (CSS, JavaScript, images)	Baiduspider-render

Baidu’s FAQ in Chinese gives the full user agent strings. For desktop search:

Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)

For mobile search:

Mozilla/5.0 (Linux;u;Android 4.2.2;zh-cn;) AppleWebKit/534.46 (KHTML,like Gecko) Version/5.1 Mobile Safari/10600.6.3 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)

Baidu identifies desktop crawling by “Baiduspider/2.0”, and mobile crawling by “Android” and “Mobile” next to “Baiduspider/2.0”. The address in these user agents, www.baidu.com/search/spider.html, now redirects to Baidu’s help center, where the FAQ answers the common questions. When you filter your logs, match these keywords rather than the full string, then verify the visit as explained below.

What Is Baiduspider-render?

Baiduspider-render is the agent Baidu uses to load the CSS, JavaScript and images of a page, so that it understands the page as a visitor sees it. Baidu announced it in March 2017 and asked site owners not to block it. In June 2017, Baidu rolled it out to all mobile crawling, while desktop crawling was not fully covered yet. Its user agents are:

Mozilla/5.0 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)

Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)

How to Verify That a Visit Comes From Baiduspider

Anyone can copy the Baiduspider user agent, and some scrapers do. Baidu recommends combining a reverse DNS lookup with a forward DNS lookup on the IP address of the visit:

1. Run a reverse lookup on the IP address: host followed by the IP on Linux, nslookup on Windows, dig -x on macOS.
2. Check the hostname: a real Baiduspider hostname ends in .baidu.com or .baidu.jp, for example baiduspider-123-125-66-120.crawl.baidu.com. Any other hostname is fake.
3. Run a forward lookup on that hostname, and check that it returns the IP address of the visit.

Baidu’s IP addresses change over time, so a fixed list goes out of date. The first version of this article, in 2019, listed these Baiduspider IP addresses as examples. Treat them as history, and verify each visit with the DNS lookups instead:

220.181.32.11   220.181.32.16   220.181.32.22   220.181.32.49
220.181.32.51   220.181.32.64   220.181.32.68   220.181.32.98
220.181.50.207  220.181.50.220  61.135.168.131  61.135.168.14
61.135.168.173  61.135.168.39

Should You Allow Baiduspider to Crawl Your Site?

If you are targeting the Chinese market, you must absolutely let Baiduspider crawl your website. You also need to follow Baidu’s basic guidelines so that your pages are indexed properly, which the next section covers.

If you are not interested in the Chinese market, you can prevent Baiduspider from crawling your website to save bandwidth for your real users. Check first that the traffic really comes from Baidu: Baidu states that its normal crawling does not congest a website’s bandwidth, and that such problems are usually caused by bots pretending to be Baiduspider.

How to Leverage Baiduspider to Rank Higher on Baidu

SEO campaigns in China must comply with Baidu’s requirements, which are like Google’s standards in some ways and different in many others.

Speed and hosting. Websites hosted overseas often perform poorly in China because of the Great Firewall: pages load slowly, or not at all. As in the rest of the world, a slow website is penalized both by search engines and by users. Since its Lightning algorithm of October 2017, Baidu favors mobile pages whose first screen loads within two seconds and demotes those that take three seconds or more. Among its recommendations, Baidu suggests a Content Delivery Network (CDN), which serves your content from the server closest to each user.

Homepage and fresh content. Baidu focuses significantly on the homepage, whereas internal pages matter more for Google. On Baidu, fresh content also counts more than long, in-depth articles.

Content in the HTML. Baiduspider-render can load JavaScript, but when it rolled out in 2017, Baidu still advised against loading valuable content with JavaScript. Keep your headings, text and links in the HTML. Flash is no longer an option at all: Adobe stopped supporting Flash Player on December 31, 2020.

These are just some of the ways to rank higher on Baidu. To boost your website’s performance in China, read our extensive guide on how to do SEO in China.

Can Baiduspider Understand the Content of Your Website?

Now that you know how Baiduspider works, you might wonder whether it can fetch, understand and render all the elements of your site. This is a crucial point: Baidu can only rank what it sees.

Baidu has its own search console, Baidu Webmaster Tools, now called the Baidu Search Resource Platform. Once your site is verified, its crawl diagnosis tool (抓取诊断) fetches a page with Baidu’s desktop or mobile user agent and shows the content Baiduspider receives. You can use it to spot content that only loads with JavaScript, hidden links added by a hacker, or an outdated IP address for your site. The crawl errors report and the robots tool of the platform complete the check.

How to Block Baiduspider

The main reason to block Baiduspider is that you are not interested in the Chinese market: you avoid wasting bandwidth that your visitors could use, and you do not slow down your website for nothing.

You may also want to keep certain pages out of Baidu, such as author pages or tags. In this case, block Baiduspider from these specific pages only.

To prevent Baiduspider from crawling your website, use a robots.txt file at the root of your website. Baiduspider follows the robots.txt protocol, and you can block it across the whole website or on selected pages only. These are the examples from Baidu’s FAQ.

To prevent Baidu from crawling the whole website:

User-agent: Baiduspider
Disallow: /

To block all crawling from Baidu except its image crawler in the /image/ directory:

User-agent: Baiduspider
Disallow: /

User-agent: Baiduspider-image
Allow: /image/

Since its robots.txt update of September 2018, Baiduspider also supports the * and $ wildcards, and it matches the paths you write exactly, upper and lower case included.

Keep a few points in mind:

• Timing. Baidu says a new robots.txt file usually takes effect within about 48 hours, and pages it has already indexed can take several months to disappear from its results.
• Advertising agents. Baiduspider-cpro and Baiduspider-ads do not follow robots.txt: they carry out what Baidu has agreed with its advertising customers, and the pages they fetch are not added to the index.
• Crawl frequency. If Baiduspider visits a site you want indexed too often, lower its crawl rate with the crawl frequency tool (抓取频次) of the Baidu Search Resource Platform rather than blocking it, and report abnormal crawling to Baidu with your server logs.
• Snapshots. To let Baidu index a page without showing a cached copy (snapshot) of it in the results, add this meta tag to the page: <meta name="Baiduspider" content="noarchive">. Baidu says the change can take two to four weeks to apply.

The standard <meta name="robots" content="noindex"> tag is the usual way to ask search engines not to index a page. For Baidu, robots.txt is the method its FAQ documents.

As a last resort, you can block the IP addresses of Baiduspider on your server. This takes time, since Baidu uses many IP addresses and they change: verify them with the DNS lookups above.

What Should You Do Next?

Understanding how Baidu crawls your website helps you decide which pages you want indexed and which you do not. Knowing which elements Baiduspider inspects, and how it reads them, also helps you spend your resources where they improve your positions in Baidu’s results.

If Baidu matters for your business, start with the basics: check that real Baiduspider visits reach your pages, that your robots.txt file lets them in, and that your key content is in the HTML. Then submit your site in Baidu Webmaster Tools and follow its crawl reports.

QPSoftware is a digital agency in Shanghai, created in 2009. Our Baidu SEO services cover the SEO audit of your website, its configuration in Baidu Webmaster Tools and its indexing. Contact us today: we can help you optimize your website for Baidu, create social media campaigns in China and use the marketing tools the Chinese market requires.

Frequently Asked Questions