How to Allow a Bot to Scrape Your WordPress Website

Overview

WordPress sites can control bot access through the robots.txt file. This guide walks you through allowing specific bots to crawl and scrape your WordPress site.

Methods for Configuring robots.txt

There are three main approaches to managing your robots.txt file in WordPress: using a plugin, editing directly via FTP, or using your hosting control panel.

Method 1: Using an SEO Plugin (Recommended)

Using Yoast SEO

Yoast SEO is one of the most popular WordPress SEO plugins and includes robots.txt editing functionality.

1. Install and activate Yoast SEO from the WordPress plugin directory

2. Navigate to SEO → Tools in your WordPress admin dashboard

3. Click on the "File Editor" option

4. Select the "robots.txt" tab

5. Add your user agent rules to the file

Example configuration:

User-agent: ChatGPT-User

Allow: /

User-agent: GPTBot

Allow: /

User-agent: *

Disallow: /wp-admin/

Allow: /wp-admin/admin-ajax.php

6. Click "Save changes to robots.txt"

Using Rank Math

Rank Math is another powerful SEO plugin with robots.txt management:

1. Install and activate Rank Math from the WordPress plugin directory

2. Go to Rank Math → General Settings

3. Click on the "Edit robots.txt" tab

4. Add your user agent directives

5. Save your changes

Method 2: Manual FTP/File Manager Access

If you prefer direct file access:

1. Connect to your site via FTP using FileZilla, Cyberduck, or your hosting's file manager

2. Navigate to your WordPress root directory (usually public_html or www)

3. Look for the robots.txt file. If it doesn't exist, create a new file named robots.txt

4. Edit the file and add your user agent rules

5. Save the file and upload it back to your server if needed

Method 3: Using Your Hosting Control Panel

Many hosting providers like SiteGround, Bluehost, or WP Engine offer robots.txt editing through their control panels:

1. Log into your hosting control panel (cPanel, Plesk, or custom dashboard)

2. Navigate to the File Manager

3. Find your WordPress installation directory

4. Create or edit the robots.txt file

5. Add your bot allowance rules

6. Save the changes

Example: Allowing ChatRankBot

To allow ChatRankBot to crawl your WordPress site, add the following to your robots.txt:

User-agent: ChatRankBot/1.0 (+https://chatrank.ai/bot)

Allow: /

Complete Example Configuration

Here's a full robots.txt example with multiple bots and WordPress-specific protections:

User-agent: ChatRankBot/1.0 (+https://chatrank.ai/bot)

Allow: /

User-agent: ChatGPT-User

Allow: /

User-agent: GPTBot

Allow: /

User-agent: Googlebot

Allow: /

User-agent: *

Disallow: /wp-admin/

Allow: /wp-admin/admin-ajax.php

Disallow: /wp-includes/

Disallow: /wp-content/plugins/

Disallow: /wp-content/cache/

Disallow: /wp-content/themes/

Disallow: /trackback/

Disallow: /feed/

Disallow: /comments/

Disallow: /category/*/*

Disallow: */trackback/

Disallow: */feed/

Disallow: */comments/

Avoiding Rate Limiting During Scraping

WordPress sites can experience performance issues if bots crawl too aggressively. Here's how to prevent rate limiting problems:

Configure Crawl-delay in robots.txt

Add a crawl-delay directive to specify how long bots should wait between requests:

User-agent: ChatRankBot/1.0 (+https://chatrank.ai/bot)

Allow: /

Crawl-delay: 3

This tells the bot to wait 3 seconds between requests.

WordPress-Specific Rate Limiting Considerations

Server Resources: WordPress sites, especially those on shared hosting, have limited resources. A crawl-delay of 3-10 seconds is typically safe.

Caching Plugins: If you use caching plugins like WP Super Cache, W3 Total Cache, or WP Rocket, your site can handle faster crawl rates:

User-agent: ChatRankBot/1.0 (+https://chatrank.ai/bot)

Allow: /

Crawl-delay: 2

Hosting Plan Considerations: Different crawl-delay recommendations by hosting type:

- Shared hosting: 5-10 seconds

- VPS/Cloud: 2-5 seconds

- Dedicated server: 1-3 seconds

- Managed WordPress (WP Engine, Kinsta): 1-2 seconds

Set Different Delays for Different Bots

User-agent: ChatRankBot/1.0 (+https://chatrank.ai/bot)

Allow: /

Crawl-delay: 3

User-agent: GPTBot

Allow: /

Crawl-delay: 5

User-agent: Googlebot

Allow: /

Crawl-delay: 2

User-agent: *

Crawl-delay: 10

Use a Rate Limiting Plugin

For more advanced control, install a rate limiting plugin:

WP Limit Login Attempts or Wordfence Security can help manage bot traffic and prevent server overload.

Monitor Server Load

Use hosting dashboard tools or plugins like Query Monitor to track:

- CPU usage during bot crawls

- Memory consumption

- Database query load

- Response times

Adjust crawl-delay values if you notice performance degradation.

Server-Level Rate Limiting (Advanced)

For more control, configure rate limiting at the server level:

For Apache (.htaccess):

apache

<IfModule mod_ratelimit.c>

    SetOutputFilter RATE_LIMIT

    SetEnv rate-limit 400

</IfModule>

For Nginx: Contact your hosting provider or add rate limiting to your Nginx configuration.

Contact Bot Operators

If a bot is crawling too aggressively despite your robots.txt settings, contact them through the URL provided in their user agent string (like https://chatrank.ai/bot for ChatRankBot).

WordPress-Specific Considerations

By default, WordPress blocks search engines if you have "Discourage search engines from indexing this site" enabled in Settings → Reading. Make sure this option is unchecked if you want bots to access your site.

Testing Your Configuration

After making changes:

1. Visit https://yoursite.com/robots.txt to verify the file is accessible

2. Use Google Search Console's robots.txt Tester tool to validate syntax

3. Wait 24-48 hours for bots to recognize the changes

4. Monitor your server logs to confirm bots are respecting crawl-delay settings

Allowing Specific Directories

To allow bots to access specific directories while restricting others:

User-agent: ChatRankBot/1.0 (+https://chatrank.ai/bot)

Allow: /blog/

Allow: /products/

Disallow: /private/

Crawl-delay: 3

User-agent: *

Disallow: /wp-admin/

Complete Example with Rate Limiting

Here's a comprehensive robots.txt for a WordPress site with optimal rate limiting:

# ChatRankBot with moderate delay for content discovery

User-agent: ChatRankBot/1.0 (+https://chatrank.ai/bot)

Allow: /

Crawl-delay: 3

# AI crawlers with slightly longer delay

User-agent: ChatGPT-User

Allow: /

Crawl-delay: 4

User-agent: GPTBot

Allow: /

Crawl-delay: 4

# Search engines with shorter delay (they're usually well-behaved)

User-agent: Googlebot

Allow: /

Crawl-delay: 2

User-agent: Bingbot

Allow: /

Crawl-delay: 2

# Default for all bots

User-agent: *

Allow: /

Crawl-delay: 5

Disallow: /wp-admin/

Allow: /wp-admin/admin-ajax.php

Disallow: /wp-includes/

Disallow: /wp-content/plugins/

Disallow: /wp-content/cache/

Disallow: /wp-content/themes/

Disallow: /trackback/

Disallow: /feed/

Disallow: /comments/

Sitemap: https://yoursite.com/wp-sitemap.xml

Common Issues

If your changes aren't appearing, check whether a caching plugin is serving an old version of robots.txt. Clear your cache after making changes.

If bots are still overwhelming your server:

1. Increase crawl-delay values

2. Check server logs to identify problematic bots

3. Consider temporarily blocking aggressive bots

4. Upgrade your hosting plan if resource limits are consistently hit

5. Implement server-level rate limiting