User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php ### https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt ### The general format is ":<#optional-comment>". ### Example = Disallow: /tag/*/ Disallows all > like so /1/2/3/4/5 ### Example = call for /tag URI with Disallow: /tag/ directive WON'T Disallow so if that's the case you have to either use * or give 2 seperate directives. Recommended for better control should be to use both and NOT the * like so /tag /tag/ /tag/*/ ### The path is case-sensitive ### The URL for the robots.txt file is - like other URLs - case-sensitive ### * designates 0 or more instances of any valid character and $ designates the end of the URL ### WARNING if you use this directive = User-agent: * which means ALL and then Disallow: / (that brackets basically tells Google to NOT crawl your site, so don't NEVER USE IT with Disallow: / accidently ### Definitions: / any valid URL (Root, URI, ALL within URI if it's a folder ### Definitions: /* same as / asterisk character is ignored ### IMPORTANT: think of directing Googlebot folder by folder bases User-agent: Googlebot Disallow: /cgi-bin Disallow: /wp-admin$ Disallow: /wp-admin*/ Disallow: /wp-includes$ Disallow: /author/admin$ Disallow: /author/admin/ Disallow: /wp-content/cache*/ Disallow: /archives$ Disallow: /archives/ # don't use below directive (for category based archives) if you already have any Pages/Posts with keywords "category" in the URL (which is indexed already by Google). For example: "Top_10_Music_category_page_of_2016.php" would match this directive thus NOT allow Google to crawl that Page/Post -- If your WordPress site has categories already indexed. Then make sure you do NOT use this directive before using 301 redirection. Watch https://www.youtube.com/watch?v=gycYWJciBG4 ###Disallow: /category$ ###Disallow: /category/ Disallow: /category/uncategorized$ Disallow: /category/uncategorized/ Disallow: /tag$ Disallow: /tag/ # don't use below directive if you already have any Pages/Posts with keywords "page" in the URL (which is indexed already by Google). For example: "Top_10_Music_category_page_of_2016.php" would match this directive thus NOT allow Google to crawl that Page/Post ###Disallow: /page$ ###Disallow: */page/ ###Disallow: /page/ Disallow: /feed$ Disallow: /feed/ Disallow: /comments/feed Disallow: /comments/feed/ Disallow: /xmlrpc.php$ Disallow: */trackback/ Disallow: /2015$ Disallow: /2016$ Disallow: */201*/ Disallow: /print$ Disallow: /print/ Disallow: /search$ Disallow: /search/ # don't use below directive if you have an Ecommerce type site with products etc. ###Disallow: /*? ### to make sure Google can calculate your content (including for mobile friendliness) it may not see some pages. Simply copy all the URL's that is blocked for Google and simply use "Allow directive" for quick fix. Or you could just leave Googlebot to crawl all css/js file formats (not recommended) ### https://developers.google.com/speed/docs/insights/BlockingJS Allow: /wp-content/uploads/ Allow: /*.js Allow: /*.css User-agent: Googlebot-Image Allow: /wp-content/uploads/ Disallow: User-agent: Googlebot-News Disallow: User-agent: Googlebot-Video Disallow: User-agent: Mediapartners-Google Disallow: User-agent: Adsbot-Google Disallow: User-agent: * Disallow: /cgi-bin Disallow: /wp-admin$ Disallow: /wp-admin*/ Disallow: /wp-includes$ Disallow: /author/admin$ Disallow: /author/admin/ Disallow: /wp-content/cache*/ Disallow: /archives$ Disallow: /archives/ # don't use below directive (for category based archives) if you already have any Pages/Posts with keywords "category" in the URL (which is indexed already by Google). For example: "Top_10_Music_category_page_of_2016.php" would match this directive thus NOT allow Google to crawl that Page/Post -- If your WordPress site has categories already indexed. Then make sure you do NOT use this directive before using 301 redirection. Watch https://www.youtube.com/watch?v=gycYWJciBG4 ###Disallow: /category$ ###Disallow: /category/ Disallow: /category/uncategorized$ Disallow: /category/uncategorized/ Disallow: /tag$ Disallow: /tag/ # don't use below directive if you already have any Pages/Posts with keywords "page" in the URL (which is indexed already by Google). For example: "Top_10_Music_category_page_of_2016.php" would match this directive thus NOT allow Google to crawl that Page/Post ###Disallow: /page$ ###Disallow: */page/ ###Disallow: /page/ Disallow: /feed$ Disallow: /feed/ Disallow: /comments/feed Disallow: /comments/feed/ Disallow: /xmlrpc.php$ Disallow: */trackback/ Disallow: /2015$ Disallow: /2016$ Disallow: */201*/ Disallow: /print$ Disallow: /print/ Disallow: /search$ Disallow: /search/ # don't use below directive if you have an Ecommerce type site with products etc. ###Disallow: /*? ### to make sure other user-agents who support Allow directive Allow: /wp-content/uploads/ Allow: /*.js Allow: /*.css # this is not smart to let crawl your site because then competitors can see your online footprint. Also, Google may see those URIs on wayback machine. But surely it is up to you! Simply uncomment if you want to use it ###User-agent: ia_archiver ###Disallow: / #If you are using WPseo by Yoast which created these sitemap URL's basically give absolute URI path to your XML sitemap. Note: you can even give your HTML sitemap URL here and you should not only have one but also declare it here. Sitemap: http://www.capitalkitchens.com.au/sitemap.xml Sitemap: http://www.capitalkitchens.com.au/post-sitemap.xml Sitemap: http://www.capitalkitchens.com.au/page-sitemap.xml Sitemap: http://www.capitalkitchens.com.au/sitemap.html ### don't forget to delete the comments before upload. Also check the whole page with Google Webmaster Tools > Crawl > robots.txt file tester (copy and paste)