Search Engines are, without doubt, the workhorses of the internet age. By indexing all the content that is on the internet, they enable millions of people to find whatever it is that they are looking for on a daily basis. Usually, this is a good thing. As a webmaster or website owner, you want to have your site crawled by Search Engines as often as is possible. This provides you with the kind of visibility that is needed to improve your rankings on Search Engine Result Pages. So, why would you wish to stop search engines from crawling your website?
When you don’t want to have search engines crawl your web pages
There are instances when you don’t wish to have search engines, such as Google, Yandex and Bing, index your web pages. Should that be the case, there are a couple of things that you can do to stop search engines from crawling your site;
Disable crawling by search engines in WordPress
The good news is that you don’t actually need to use third party plugins to stop search engines from indexing your WordPress website. WordPress comes with a build in feature that stops Google and other search engines from crawling your site. To enable it, all that you need to do is the following;
- Login into the admin area of your WordPress website.
- Go to Settings, and then click on “Reading.”
- Scroll down to “Search Engine Visibility.”
- Make sure that the “Discourage search engines from indexing this site” is selected.
You will notice that there is an notice in the above screenshot telling you that it’s actually up to Search Engines whether or not to index your website. However, once you have saved your chances, your site will no longer be actively crawled by Search Engine bots.
Method 2 for stopping search engines from crawling your WordPress Website
If you are tech savvy, you can also discourage Search Engine bots from crawling your site by making changes to the robots.txt file in your website’s files. Use your FTP client to get access to your website’s files. Robost.txt will usually be located in the public_html folder inside the “WordPress folder.” Once you have located the robots.txt file, enter the following syntax;
User-agent: *
Disallow: /
That’s it. Now search engines will no longer be able to index your website.
Why disable access by Search Engines.
There are a number of reasons why you may wish to disable access to your site by Search Engines;
- You are making changes to your site and do not wish to have search engines crawl at that particular time.
- You wish to keep your site private.
Whatever your reasons, following the steps that we have outlined above will stop search engines from indexing the content that’s on your website.
