Latest Technology News,Laptop&PC Reviews,Mobile Gadgets,how to,help me,free browsing cheat,hack,tricks,blogging,do it for me, HOW TOS etc.

Advertise with us

Follow by Email


Lefkofsky Talks About the Need for More Empathy

Lefkofsky Talks About the Need for More Empathy People who have success in the realm of business innovation, like Eric Lefkofsky, may ...

Robot.txt for Blog SEO: List of Generator and Checker tool

Robot.txt for Blog SEO: List of Generator and Checker tool

Robot.txt file is simply a line of codes that tells crawler what part of site to index and which not. As a blogger you should at least know the basic of Robot.txt file. Although it sounds more technical word but it is just as simple as publishing post. 

In this tutorial I will be sharing about tools that will help you to easily create Robot.txt file and tools to check whether the file you created is right or not. Because just a line of wrong code can block your site from being index in Google which ultimately leads to no traffic.
You don't need to go into the technical aspects and meaning of each line and neither I am going to teach you, but I will be sharing simple meaning and a tool that makes it easy to handle all.

Advantage of using Robot.txt

As I said with Robot.txt file you can tell Google to not index certain URL or part of the website, therefore you can take advantage in many ways
  • For example there are many parts of blog like archive, labels, search pages which we usually don't like to display in search results. All these pages contains duplicate content in perspective of a crawler which in returns cause duplicate content issue and finally lower ranking.
  • You will get the notification for the same in webmasters tool. To fix that I already shared the ways previously.
Fix Duplicate Meta Description Error in Webmaster tool
How to Remove 404 Broken Links Errors
  • So by simply blocking all these type of pages in Robot.txt file you will save all these kind of issues.
  • Similarly on WordPress you can block wp-admin area from robots and similarly other directories.
  • Having a Robot.txt file gives you advantage of a better blog management in terms of seo and Google crawler love that too.

Disadvantages of Robot.txt

There are no disadvantages as such but wrong use of Robot.txt file could lead to serious problem in blog.
  • For example you could accidentally place a code that actually tells crawler to not index you blog.
  • Similarly you might end up yourself blocking important pages.
  • When I was newbie I also blocked my sitemap from Google crawler and that caused the problem of not indexing my further posts. But I figured out in webmasters tool and immediately fixed the problem.

But you don't need to worry if you make use of tools that I listed below. But using these tools you can check whether you have correctly placed the code or not.

According to google:
A robots.txt file restricts access to your site by search engine robots that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a robots.txt file exists that prevents them from accessing certain pages.

How to see your robot.txt file

  • Enter following url in the address bar

  • Replace yoursiteurl with your own blog url

The simplest robots.txt file

  • User-agent: the robot the following rule applies to
  • Disallow: the URL you want to block
User-Agent: Googlebot
Disallow: /folder2/

Tools to create Robot.txt file:

These tools are simple to use and it makes all hectic of understanding Robot.txt file easiest. 

1. SEObook tool:

  • You just need to add the URL which you want to block and just hit enters. This tools will automatically add it to you file. After you complete just copy the code and paste it into your Robot.txt file. 
  • Finally use tools to check the correctness.
Alternate tool

Add robot.txt to Blogger blog:

  • In blogger you can easily add Robot.txt code by going to settings>Search preferences.

Tools to check/Test Robot.txt file:

I strongly recommend you to use these tools after you added the code to your blog, because if something went wrong you would not be able to recognize the problem.

1. Using Google webmaster tool

  • Webmasters tool for Google is best for testing purposes and for that you can go to Crawl section of Webmaster Tools and then Blocked URLs 
  • Just copy the code from your blogger Robot.txt file and paste it in the box provided.

  • Now in the lower box enter your URL which you want to check and click test.

  • You will see a message showing the status if the URL is blocked by Robot.txt or not. It also shows the line which is allowing or blocking.

Remember adding code here in webmasters tool is just for testing purposes, it doesn't add Robot.txt file to you blog. For that you have to add the file in you blog only.

2. Frbee Robot.txt check

This was the simplest yet effective tutorial to understand and implementation of Robot.txt file in your blog. All the probloggers use Robot.txt for their blogs and you should too, but make sure you don't end up your blog harming firm it. That's why I have shared the testing tools as well. 
Do you have any questions? Did you placed wrong code in your Robot.txt file? Do you want to know what's the meaning of each line is? Please feel free to ask me in the comments below. I am always here to help you. Also share you experience of using Robot.txt file. Don't forget to like us in Facebook and Google plus.

No comments:

Post a Comment

FOLLOW US ON FACEBOOK: Ehijoshuatechblog
:::. Please dont leave here without dropping a comment and feel free to Share/Like this Tutorial to your friends outside there .:::

Popular search

Advertise with us

Popular Posts All time

Search the web

Submit Guest Post

Submit Guest Post
Hello we are EJTB we accept sponsored post

Popular Posts

Sponsored advert

Sponsored advert
Send FREE Bulk SMS Text Messages to Mobile Phones

Readers Online

Advertise here

Advertise here
I can help you promote your offline/online business, start-up, seminar etc to my targeted blog readers and you will get a good value for your money.

About Me




. The latest software update 1 free browsing cheat 1 REVIEW 32nd 3D 7 PRO APPLE BLACK 4G 5 Mobile Marketing Tips 50GB on MTN 9MOBILE actificial afrideals airtel Airtel Night Plan Data Airtel Unlimited Surfing Airtel Unlimited Surfing With N0.0 Airtel Unlimited Surfing With N0.0 Via Psiphon Alcatel Go Play 4G LTE and and HOW TO'S visit and iPhone 6s. and Text Messages and Xperia L3 leaks online Android Android 2.1 Android 2.2 Android 2019 Android 6.0 Marshmallow Android 7.0 Android games without WIFI Android IMEI to BB IMEI + HTC Androids Android Phones ANDROID TUTORIAL APPLE APPLE CLEARS UP DATA PRIVACY CONCERNS IN IOS 10 Apple has released iOS 11.0.3 Apple Watch Series 3 article ATTRACTIVE 4G LTE PLANS. babies bendable phone Best Offline Shooting Games For Android Birthday bitcoin BITTORRENT Blackberry Blogger Template BLOGGING BLOGGING TUTORIALS BlogSpot Blogging Platform Browsing Tips Bulk SMS bulksms by Andy Rubin Calculator Using HTML Caller Tunes Business Calls Camon C9 most powerful features and review Can You Really Make Money Flipping Houses? Celcom’s revised FIRST postpaid plans win some features Celebrates china Chrome cooking games for Android CSS & Javascript detail the Galaxy Note 8's rumoured specs Disable Windows Updates in Windows 10 all Versions Permanently Doogee Bigboy DG600 DroidVPN Duo ehijoshua jokes Enjoying Unlimited Free Browsing On PC etisalat etisalat BB10 unlimited surfing Etisalat Bblite Unlimted Plan Etisalat Night Data Plan etisalate facebook Facebook CEO Family Locator Apps FD-LTE bands Firefox Five Easy Ways To Save Your [Mobile & Computer] Data for iPhone for iPhone 7 For latest Technology News Free Browsing Garanntor? get a Caller tune set up for your in Nigeria Gionee M5 plus giveaway free cracked glo Glo Bis Glo Bis 3GB of 1K Glo Free Browsing Glo jollific8 Google Docs Google Pixel 2 Google Pixel 2 and Pixel 2 XL GTBank Virtual MasterCard GUST POST Hackers Health Sciences His home’s audio system Hot Glo Unlimited Free Browsing For december 2018 Hot S X521 how to How To Create A Free Blog HUAWEI HUB8.COM i1s i2 IFTTT includes bug fixes Infinix Hot S Infinix HotNote X551 Infinix Note 3 Pro X537 Infinix Note 3 X601 Infinix Zero 4 Innjoo i1 instagram Instagram Stories Install / Flash ClockworkMod Recovery Install two Facebook Apps in a Single Android Phone. No root required. Investment Plan for Retirement ios iPad iPhone 3GS/iOS 6 iphone 7 iphone 8 iPhone X iPhone X issues iPhone X vs Galaxy Note 8 IPTV jamb Jumia keyboard Laptop&PC Reviews LeEco Lefkofsky LG LG V20 LG V30 Full lose some love love message Make Free Calls Make Money From Home Make Money Gaming Online MAKING MONEY IN NAIJA MAKING MONEY ONLINE Mark Zuckerberg Metlife eService Insurance missingring MOBILE Mobile Gadgets MOBILE PHONES ORANGEVODAFONEO2EET-MOBILETHREEVIRGINIOSAPPLE IPHONE X Moto E3 MOTOROLA Mtn MTN Android One data plan MTN BBLITED MTN Data MTN Give Away Data Plan of 7GB For Just N2000 MTN Sudan MTN Unlimited new iPhone New Working Setting For Glo N0.00 Unlimited Free Browsing newbulksms newbulksms nigeria ng nigeria Nigeria: Airtel reduces data tariffs Nintendo 64 (N64) Emulators For Android Nintendo 64 (N64) Emulators For PC Nokia 8 Nokia 8 Key Specifications NOKIA 8 SPECIFICATION NOKIA 9 COMPLETE SPECIFICATION NOKIA 9 CURVED SCREEN Nokia Edge 2017 Nokia S40 Nokia Symbian S60 ntel Launches Fastest Router In Nigeria Offline Games 2019 Offline RPG Games for Android OgbongeBlog Blogger Template ONEPLUS OnePlus 5T Online Transactions Pepsi P1S 4G perfect Android phone phone PHONES Plus Pokemon Go ps4 QUESTRA. INVEST TODAY! Razer READ MORE ABOUT relationship REVIEWS Root an Android Device Rooting Samsung Galaxy Note 4 Safari SamMobile SAMSUNG Samsung Galaxy J7 + plus Samsung Galaxy Note 4 Samsung Galaxy Note 7 Specs & Price Samsung Galaxy Note 8 Samsung Galaxy Note 8 release date Samsung Galaxy S10 Samsung's UI screenshot on your Mac Seven Things Your ATM Card Can Do Social network Sony Xperia 1 Sponsored Study in Cyprus SUBMIT GUEST POST TD-LTE TECNO CAMON C9 Template the easy way The Essential Phone The Nokia 8 will get a bigger brother dubbed the Nokia 9 - Nokia 9 Release STILL Happening; It's a Phablet Version of Nokia 8 (check it out like i comment and share ) The Prices Of The Newly Launched Gionee M6 And M6 Plus Will Surprise You TIPS to Block Spam Top 10 Best Web Browsers For PC Top 20 Apps Tweak IMEI of Infinix Hot X507 Twitter unity bank plc Unlimited Free Browsing Unlimited Free Data Bundle vpn web hosting What Is Sonos? Whatsapp Whatsapp to Disconnect Symbian OS from the Whataapp network by December Whiteboard Sketch Video Whogohost Why Sonos? Wiko Wiko Bloom Windows 10 marks 1 year today and free upgrade expired Windows Phone 7.1 WordPress WORLDPRESS Write For Us Xiaomi Mi A1 Android One Specs Xiaomi Mi Note 2 Pro Xiaomi Redmi Note 5A Xperia 10 Xperia 10 Plus XPERIA Cars Road Trip YES 4G LTE PREPAID YOUR OPINION!! Which Nigerian Internet Service Provider Has The Most Friendly & Cheap Data Plans? YouTube zain