Comments on: iRobots.txt SEO http://markbeljaars.com An exercise in shameless self promotion Mon, 27 Dec 2010 09:14:59 +0000 hourly 1 http://wordpress.org/?v= By: WordPress Plugins from A to Z Podcast – I for Internal Linking and iRobots.txt http://markbeljaars.com/plugins/irobotstxt-seo/comment-page-2/#comment-19318 WordPress Plugins from A to Z Podcast – I for Internal Linking and iRobots.txt Tue, 07 Dec 2010 05:52:00 +0000 http://markbeljaars.com/?page_id=320#comment-19318 [...] is upon us today. What I will be covering today is the letter I for CF Internal Link Shortcode & iRobots.txt plug-ins. These plug-in are pretty straight forward which is why there is two here [...] [...] is upon us today. What I will be covering today is the letter I for CF Internal Link Shortcode & iRobots.txt plug-ins. These plug-in are pretty straight forward which is why there is two here [...]

]]> By: Mark http://markbeljaars.com/plugins/irobotstxt-seo/comment-page-1/#comment-5010 Mark Sun, 04 Jul 2010 11:31:32 +0000 http://markbeljaars.com/?page_id=320#comment-5010 Hi Nick, The only thing Google XML Sitemaps does is add a single line to the end of the robots.txt file pointing to the XML sitemap. If you disable XML sitemaps from touching the robots.txt file (by deselecting "Add sitemap URL to the virtual robots.txt file. " within the XML-Sitemaps settings), then iRobots will automatically add this entry for you if it detects that Google XML Sitemaps plugin is installed. Hope this helps, Mark. Hi Nick,
The only thing Google XML Sitemaps does is add a single line to the end of the robots.txt file pointing to the XML sitemap. If you disable XML sitemaps from touching the robots.txt file (by deselecting “Add sitemap URL to the virtual robots.txt file. ” within the XML-Sitemaps settings), then iRobots will automatically add this entry for you if it detects that Google XML Sitemaps plugin is installed.
Hope this helps,
Mark.

]]> By: Nick http://markbeljaars.com/plugins/irobotstxt-seo/comment-page-1/#comment-5005 Nick Sun, 04 Jul 2010 09:31:00 +0000 http://markbeljaars.com/?page_id=320#comment-5005 Hi I'm a little confused about how iRobots.txt SEO works when the Google XML Sitemaps plugin is installed. Should the iRobots plugin be installed after the Google XML Sitemaps plugin to enable the Sitemaps plugin to be detected? Is there a potential for conflict which requires the settings in one of the two plugins to be changed? If which settings in which plugin? I would be grateful for any clarification. Thanks Hi I’m a little confused about how iRobots.txt SEO works when the Google XML Sitemaps plugin is installed.

Should the iRobots plugin be installed after the Google XML Sitemaps plugin to enable the Sitemaps plugin to be detected?

Is there a potential for conflict which requires the settings in one of the two plugins to be changed? If which settings in which plugin?

I would be grateful for any clarification.
Thanks

]]> By: Best seo wordpress plugins http://markbeljaars.com/plugins/irobotstxt-seo/comment-page-1/#comment-1842 Best seo wordpress plugins Fri, 23 Apr 2010 08:50:53 +0000 http://markbeljaars.com/?page_id=320#comment-1842 [...] iRobots.txt SEO : is a fully customizable robots.txt virtual file generator. IRSEO creates a highly optimized and secure robots.txt file straight out of the box. Users may choose to enable or disable specific user agents, directories or files using intuitive options all of which include detailed instructions. [...] [...] iRobots.txt SEO : is a fully customizable robots.txt virtual file generator. IRSEO creates a highly optimized and secure robots.txt file straight out of the box. Users may choose to enable or disable specific user agents, directories or files using intuitive options all of which include detailed instructions. [...]

]]> By: Mark http://markbeljaars.com/plugins/irobotstxt-seo/comment-page-1/#comment-1424 Mark Fri, 02 Apr 2010 08:20:13 +0000 http://markbeljaars.com/?page_id=320#comment-1424 Hi JoJo. I uses nonces as an added security measure to stop XSS attacks from modifying the robots.txt file. As a quick fix, you may be able to edit the plugin PHP file and delete the following text (it appears several times throughout the code)... && check_admin_referer('irseo-change-options-nonce') This will disable nonces. I find this really strange though as nonces tend to work well in Wordpress. Please let me know if this "fixes" your problem. Thank you, Mark. Hi JoJo. I uses nonces as an added security measure to stop XSS attacks from modifying the robots.txt file. As a quick fix, you may be able to edit the plugin PHP file and delete the following text (it appears several times throughout the code)…

&& check_admin_referer(‘irseo-change-options-nonce’)

This will disable nonces. I find this really strange though as nonces tend to work well in WordPress. Please let me know if this “fixes” your problem.

Thank you,
Mark.

]]> By: jojo http://markbeljaars.com/plugins/irobotstxt-seo/comment-page-1/#comment-1393 jojo Wed, 31 Mar 2010 03:06:48 +0000 http://markbeljaars.com/?page_id=320#comment-1393 Hi, I have the plugin installed and mysite.com/robots.txt shows up fine, but when I click on the plugin settings, I get a Wordpress error page that simply says, "You do not have sufficient permissions to access this page." Any ideas? Hi, I have the plugin installed and mysite.com/robots.txt shows up fine, but when I click on the plugin settings, I get a WordPress error page that simply says, “You do not have sufficient permissions to access this page.”

Any ideas?

]]> By: Mark http://markbeljaars.com/plugins/irobotstxt-seo/comment-page-1/#comment-1130 Mark Sat, 13 Mar 2010 06:02:43 +0000 http://markbeljaars.com/?page_id=320#comment-1130 Sorry for the late reply. Akismet had marked your post as spam due to the multiple links and I only check the spam emails once a week or so before I delete them. I checked out your robots.txt file and it looks OK. In webmaster tools, under site configuration/crawler access, select test to test your site for the Googlebot user agent. I tested your exact robots.txt file and it says googlebot is allowed to crawl the site. If you suspect the plugin is causing you problems, de-activate the plugin and see whether the problem still remains. It is perfectly OK to not have a robots.txt file, however Google will index all your admin pages etc. Restricted access may also be caused by your .htaccess file. Mark. Sorry for the late reply. Akismet had marked your post as spam due to the multiple links and I only check the spam emails once a week or so before I delete them.

I checked out your robots.txt file and it looks OK. In webmaster tools, under site configuration/crawler access, select test to test your site for the Googlebot user agent. I tested your exact robots.txt file and it says googlebot is allowed to crawl the site.

If you suspect the plugin is causing you problems, de-activate the plugin and see whether the problem still remains. It is perfectly OK to not have a robots.txt file, however Google will index all your admin pages etc.

Restricted access may also be caused by your .htaccess file.

Mark.

]]> By: Ali http://markbeljaars.com/plugins/irobotstxt-seo/comment-page-1/#comment-1123 Ali Fri, 12 Mar 2010 00:09:35 +0000 http://markbeljaars.com/?page_id=320#comment-1123 Hi i have installed your plug in and activate it but search engines are not crawling on my site any page what is the problem. i use Google web master tool i add up my sitemap but the msg appears that we are restricted by the robot.txt. plz help me. My link for robots and sitemap is http://reviewaccounting.com/robots.txt http://reviewaccounting.com/sitemap.xml.gz plz tell me what to do i am waiting for your reply Hi i have installed your plug in and activate it but search engines are not crawling on my site any page what is the problem. i use Google web master tool i add up my sitemap but the msg appears that we are restricted by the robot.txt. plz help me. My link for robots and sitemap is
http://reviewaccounting.com/robots.txt
http://reviewaccounting.com/sitemap.xml.gz
plz tell me what to do

i am waiting for your reply

]]> By: Mark http://markbeljaars.com/plugins/irobotstxt-seo/comment-page-1/#comment-1087 Mark Fri, 05 Mar 2010 11:56:03 +0000 http://markbeljaars.com/?page_id=320#comment-1087 Hi Natalie, Can you explain this a little more? Do you have iRobots.txt options in the admin settings pane? You can check whether the plugin is working by typing the following into a browser: http://yoururl/robots.txt For example, my site's robots.txt file is located at: <a href="http://markbeljaars.com/robots.txt" rel="nofollow">http://markbeljaars.com/robots.txt</a>. If the robots.txt file does not exist, there could be three possible reasons: 1. The plugin failed to install for some reason. Remove the previous install then try downloading from my site. Then select plugins/add new from the admin panel and click the upload link. Upload the zip file and activate. 2. Your theme is overriding virtual file requests. 3. Your .htaccess file is blocking the robots.txt file. Mark. Hi Natalie,

Can you explain this a little more? Do you have iRobots.txt options in the admin settings pane? You can check whether the plugin is working by typing the following into a browser:

http://yoururl/robots.txt

For example, my site’s robots.txt file is located at: http://markbeljaars.com/robots.txt.

If the robots.txt file does not exist, there could be three possible reasons:

1. The plugin failed to install for some reason. Remove the previous install then try downloading from my site. Then select plugins/add new from the admin panel and click the upload link. Upload the zip file and activate.
2. Your theme is overriding virtual file requests.
3. Your .htaccess file is blocking the robots.txt file.

Mark.

]]> By: Natalie http://markbeljaars.com/plugins/irobotstxt-seo/comment-page-1/#comment-1075 Natalie Wed, 03 Mar 2010 18:22:36 +0000 http://markbeljaars.com/?page_id=320#comment-1075 Hi mark, I have installed your plug in but can not see it anywhere in the source. If I don't see it does that mean it is not working? Hi mark,

I have installed your plug in but can not see it anywhere in the source.

If I don’t see it does that mean it is not working?

]]> By: Mark http://markbeljaars.com/plugins/irobotstxt-seo/comment-page-1/#comment-984 Mark Thu, 18 Feb 2010 12:56:17 +0000 http://markbeljaars.com/?page_id=320#comment-984 Hi Andy, I just released an update that fixes this bug and adds a new option to disallow duplicate content. Hopefully it works OK for you now. Mark. Hi Andy,
I just released an update that fixes this bug and adds a new option to disallow duplicate content. Hopefully it works OK for you now.
Mark.

]]> By: Mark http://markbeljaars.com/plugins/irobotstxt-seo/comment-page-1/#comment-934 Mark Thu, 11 Feb 2010 00:27:46 +0000 http://markbeljaars.com/?page_id=320#comment-934 Hi Andy, I haven't seen this error before. What version of PHP are you running? If you are happy editing PHP files, you can simply delete this line and the plugin should start working (in PHP, files are closed automatically if the programmer forgets to close them). In any case, I will fix the code tonight (Australian time) and upload a new revision. Mark. Hi Andy,
I haven’t seen this error before. What version of PHP are you running? If you are happy editing PHP files, you can simply delete this line and the plugin should start working (in PHP, files are closed automatically if the programmer forgets to close them). In any case, I will fix the code tonight (Australian time) and upload a new revision.
Mark.

]]> By: Mark http://markbeljaars.com/plugins/irobotstxt-seo/comment-page-1/#comment-933 Mark Thu, 11 Feb 2010 00:23:43 +0000 http://markbeljaars.com/?page_id=320#comment-933 Hi Jadah, The robots.txt file is actually a virtual file. The plugin detects when somebody (or somebot) is trying to display the http://[www.]website/robots.txt file and intercepts this request. The plugin then instead outputs a text stream that makes it look like the file is being sent. In this way, there is notihing you need to upload. Also, any changes to the robots.txt file are immidiately reflected. There is one catch with this method though. If a physical robots.txt file exists (ie one that you have FTP uploaded), then this will always be displayed instead of the virtual file. This means that you will need to delete (or rename) any physical robots.txt files to use this plugin. Hope this helps, Mark. Hi Jadah,
The robots.txt file is actually a virtual file. The plugin detects when somebody (or somebot) is trying to display the http://www.website/robots.txt file and intercepts this request. The plugin then instead outputs a text stream that makes it look like the file is being sent. In this way, there is notihing you need to upload. Also, any changes to the robots.txt file are immidiately reflected. There is one catch with this method though. If a physical robots.txt file exists (ie one that you have FTP uploaded), then this will always be displayed instead of the virtual file. This means that you will need to delete (or rename) any physical robots.txt files to use this plugin.
Hope this helps,
Mark.

]]> By: Andy Fragen http://markbeljaars.com/plugins/irobotstxt-seo/comment-page-1/#comment-932 Andy Fragen Wed, 10 Feb 2010 23:45:05 +0000 http://markbeljaars.com/?page_id=320#comment-932 Mark, I just installed your plugin and I get the following error. Warning: fclose(): supplied argument is not a valid stream resource in /Library/WebServer/Documents/xxxxxxx/wp-content/plugins/irobotstxt-seo/irobotstxt-seo.php on line 322 Running WP 2.9.1. Any ideas? Andy Mark,

I just installed your plugin and I get the following error.

Warning: fclose(): supplied argument is not a valid stream resource in /Library/WebServer/Documents/xxxxxxx/wp-content/plugins/irobotstxt-seo/irobotstxt-seo.php on line 322

Running WP 2.9.1. Any ideas?

Andy

]]> By: Jadah http://markbeljaars.com/plugins/irobotstxt-seo/comment-page-1/#comment-918 Jadah Tue, 09 Feb 2010 20:12:08 +0000 http://markbeljaars.com/?page_id=320#comment-918 Hi, I have one question, when I use your plugin and create the robots.txt file, does your plugin upload it to the websites directory, or do I have to copy the file and upload it through FTP.. Thanks, the best robots.txt plugin so far. Hi, I have one question, when I use your plugin and create the robots.txt file, does your plugin upload it to the websites directory, or do I have to copy the file and upload it through http://FTP..

Thanks, the best robots.txt plugin so far.

]]>