Top 5 reasons why Google may not crawl Websites

I get lots of emails where people have difficulties in getting site indexed and crawled by Google

Google is not crawling my XYZ website. I don’t know why? Can you help me please? My URL is somedomain.com

And if you post such question on any forum like one of the user of Go4Expert.com, IndianSword posted Google indexing and crawling issue? people always tend to suggest to get more incoming links for new websites and I also did the same but then after some general answers he still had problems and so I needed to see if there is something which is missing on his website. Digging his website’s HTML deeper I found it had some Meta issues for some misconfiguration settings in WordPress.

Now let’s see what could be the possible reason if your website in not in Google’s Index or if Google is not crawling your website

1. Blocked by robots.txt

We should always block certain areas of the website like admin area or test areas but doing it rightly with the right syntax is also a must. Done wrongly you can have problems with indexing in Google. See if you have any code in your robots.txt file which is blocking Googlebot to index your main pages. If you know the syntax rightly you can check yourself and if you are not aware of what should be the correct syntax add your website in Google Webmaster Tools and see if you have any wanted urls blocked by your robots.txt file.

2. Meta tags

Many webmaster by mistake have noindex Meta like

  • <meta name=”robots” content=”noindex, nofollow”>
  • <meta name=”robots” content=”noindex”>

If you have any such HTML on pages you want to be in Google Index try getting rid of such Meta.

Apart from noindex and nofollow I have also seen revisit-after Meta.

  • <meta name=”revisit-after” content=”1 day”>

Though I haven’t seen any evidence of this being supported by major search engines but still if your page says <meta name=”revisit-after” content=”1 day”>, that won’t be taken as an instruction for search engine bots to return every day but rather as an instruction to go away from your site if it hasn’t been 1 day since the last visit.

3. Site is new

New websites are crawled less often in Google than more established websites and if your website is new there are high chances that Google would not be indexing your site very often. Instead of worrying much about it in Google’s index you should focus more on core of your business i.e. if its forum or blog getting more content or if its an e-commerce making more sales because eventually Google would start on your site when you have the right kind of mix with content and incoming links. If you addicted to checking your site stats or Google index read Getting out of Recession for Blogger

4. Less update frequency

If your websites pages do not update often then there is no reason for Googlebot to visit your site and update its cache. If your website does not update content regularly then there are chances that when you update the content Googlebot would not cache the new version that often.

Now if you have a static HTML homepage or website there is no need for Googlebot to come to your site very often. Now when you do some design changes or text changes Googlebot would not be updating your cache often. To avoid such issues you can always add Blog / Forum to your static site which gets more frequently updated. Blog could be as simple as what you or your company are undertaking.

5. Low incoming links

When your site is new you are bound to have few or no incoming links to your site and so try focusing on getting some good quality links to your site and instead of checking your site in Google’s index try getting more incoming links. See how to Build Links to Website. You should never be satisfied with the number of links your website has and always think about getting few more.

The Possible reason why Google may not crawl or index your website can be endless but I have tried to list few of the reasons I have experienced over time.

Still have issues post them in comments and I would be more than happy to give it a look.

Sign up for Free Course on How to be an Online Entrepreneur?

Don't wanna sign up but prefer posts delivered to your Inbox?

Not planning to subscribe through Email. Check out the RSS Feed.

  • David

    My Website has been indexed and crawled while the information found on webmastertool.. But i didnt get much visitors from google.. I dono why.. Last week i got good visitors and suddenly it has stopped.. can any one please give solution for this… yesterday i added site in webmaster tool…

    • http://imtips.co/ Shabbir

      You may be ranking for some terms in Google that are all gone now.

      • David

        Hi, I added robot.txt file and it has been removed on the same day… But still after 2 days of removal. my page is not crawling on googlebot… i checked with webmaster tool and shows robot.txt file under my file name.. how many days will it take to refresh the website by google…

        • http://imtips.co/ Shabbir

          robots.txt file for few hours does not mean your site will not be indexed. It has to be something else David.

  • http://www.technoarea.in Rahul Sharma

    I have a site which was popular at one time, Google used to crawl my site withn few minutes of posting articles, but now it takes days to crawl, My alexa ranking is also going down.

    I cant understand what has happened because earlier my site was working very good but all of sudden it has dropped.

    my site is at http://www.technoarea.in

    • http://imtips.co/ Shabbir

      First of all you have a ccTLD domain but I assume you are targeting global audience and once of the reason could be that you are ranking good in Google India but not elsewhere and this could be one of the prime reason for your drop in traffic as Google algo now ranks pages from ccTLDs in country specific search engines.

  • http://markpulido.com Sunil

    Hi
    I am having markpulido.com, when i am checking it in webmaster tool it is showing the error that “http://www.markpulido.com/: Googlebot can’t access your site” and in the dashboard of Webmaster tool in crawl errors it is showing “Google couldn’t access your site because of a DNS error.” Please help me i have talked with my hosting provider they are saying we dont have any error from our side.
    Please help me as i am facing this problem from last 15 days.

    Sunil Kumar

    • http://imtips.co/ Shabbir

      Sunil, I don’t see any DNS issues ( http://intodns.com/markpulido.com ) and so try removing the site from GWT and add again to see if it works. I see that your site’s home page is indexed and cached by Google

  • http://www.i2z.in vicky sadhu

    My site http://www.i2z.in‘s post are not getting cached by google since last few days..any idea why this is happing??

    • http://imtips.co/ Shabbir

      I don’t see any value for the content that you have on your site and it looks like all copied and so no reason for Google to crawl your content Vicky

  • http://entertainjosh.com/ vamsi krishna

    hai,
    This is Vamsi. I started a website 2 months ago. I have a Big Problem with google. google takes my pages indexed. In Crwl errors is 0 nice. At the Same time Webmaster tools >> Health >> Index Status total indexed is 0. Ever crawled 1. Blocked by robots and Removed is 0. Whats happening. my robots.txt is well.
    # User-agent: *

    # Disallow: /wp-admin/

    # Disallow: /wp-includes/

    # Disallow: /wp-trackback

    # Disallow: /wp-feed

    # Disallow: /wp-comments

    # Disallow: /wp-content/plugins

    # Disallow: /wp-content/themes

    # Disallow: /wp-login.php

    # Disallow: /wp-register.php

    # Disallow: /feed

    # Disallow: /trackback

    # Disallow: /cgi-bin

    # Disallow: /comments

    # Disallow: *?s=

    Sitemap: http://entertainjosh.com/sitemap.xml
    Sitemap: http://entertainjosh.com/sitemap.xml

    At the same time Sub Domain is fine. Actually i Started this site in another domain. i had change the domain name and files are transferred old domain to new domain. In that site( new ) firtly i use duplicate content. After some days i removed copy content and post unique content. i had submit sitemap.xml. and google takes this. But Present in Webmaster tools health >> Index Status >> Total indexed is 0. Ever Crawled is 0. Main indexed is shows 202 pages indexed y?

    • http://imtips.co/ Shabbir

      Vamsi, I see all your content are copy pasted from other sites and so I don’t see a reason for Google to index your site because Google already have that content in their index.

  • http://www.mangalpandit.com Aditya

    Google stop crawling after 2 Nov 2012

    Previously when i do site:mangalpandit.com, it displays in google search result.

    Now it is not coming.

    Can you please help me.

    Thanks in advance.

    • http://imtips.co/ Shabbir

      Your site is very static and on top of that there is hardly any content to be crawled.

      Also one more thing could be your backlinks have either vanished or may be from pages that are not of good quality and so it could be that Google dropped your site from index as well.

  • Costin

    I have a website http://www.haineonlinetari.com/, which does not appear in searches though is indexed. Any idea what the problem is? Thanks in advance, Shabbir !

    • http://imtips.co/ Shabbir

      Costin, For what keyword searches do you expect this site to appear? I see that your site does appear for the term haine online tari which means your site would need to be focusing on getting content related to your main keywords as well as drive links to those content as well as your home page.

  • http://www.balajistirupur.in balaji

    how ot bring my site on google even faster

    u can see my website on first page 7th result on google for the keyword”balaji srinivasan”…please help me

    • http://imtips.co/ Shabbir

      You should get more links with your name as Anchor. If you post in blog comments like you have done here, You should include your full name and that would drive few links to your site with your name and would gain ranking.

  • http://geopvp.com geopvp

    why my site geopvp.com is not cached in google

    • http://imtips.co/ Shabbir

      I see the site is cached in Google?

  • Kiran

    Hi,
    I need your help, I have website http://www.torontoreitement.com/, which is not cache by google, I don’t know the reasons. Please tell me the exact error in this site.

    • http://imtips.co/ Shabbir

      Kiran, your site does not open for me.

  • http://techgyo.com Sreejesh@techgyo

    I used to have sitelinks for my site before 2 days. Now i see nothing at all, and my site is not cached in Google. Also only 165 pages are now indexed when compared to 2600 indexed before. I’m wondering what went wrong, i didn’t make any post since then and there was no major changes to the site design too. What would you think might have gone wrong? Any advice would be greatly appreciated.

    And your share buttons looks awesome, can you share me the code or the plugin you’ve used?

    • http://imtips.co/ Shabbir

      It may be one of those Google Dance and 2 days is small time frame to analyze things. Wait for few days and see if things come back to normal. Make sure you work on things recommended by Google and content.

      • http://techgyo.com Sreejesh@techgyo

        Thanks for the quick reply, I think I should wait. By the way can you name the plugin you are using for those share buttons, It would be great if you could share it with me.

        • http://imtips.co/ Shabbir

          Which Share buttons? On the right side ones or after the post?

          • http://techgyo.com Sreejesh@techgyo

            I mean the onces after the post.

            • http://techgyo.com Sreejesh@techgyo

              Please feel free to mail me the answer :)

            • http://imtips.co/ Shabbir

              It is not done using any plugin but manually done with AddThis. You can grab the html for the code if you want.

  • http://www.money9.net Pankaj

    Can you suggest something for robots.txt?
    I’m little bit confused with making robots.txt
    :(

    • http://imtips.co/ Shabbir

      What you want to know for robots.txt

      • http://www.money9.net Pankaj

        means, what lines should be there.

  • http://petuaterbaik.kaer-media.org Petua Terbaik

    i’m still have a problem google bot not updating my index at http://petuaterbaik.kaer-media.org can u tell me what wrong ?

    • http://imtips.co/ Shabbir

      Google cached your site on 29 Oct 2010 and as there is not much activity involved Google does not visit your site regularly and so waiting is your best option. Apart from that do submit the sitemap so when you add new content Google is notified of it.

  • umeshg

    Hiiiiiiii I m divine light I read this information, i really got the more suitable answer. It porvide me some good & accurate results. thank you.

  • http://imtips.co Shabbir Bhimani

    No you are absolutely right and your homepage cache was of 8th December only but there are other pages of your site as well and apply the same method on pages and posts of 14th December and you will see that its also cached which means there is nothing wrong in caching of posts and home page cache is not updated which is perfectly normal

  • adorigraphics

    Please forgive me of my ignorance, I am new to the whole SEO thing. I have sitemeter installed on my blog (http:adorigraphics.blogspot.com) and I used to notice that googlebot visited my blog maybe 2 or 3 times a day. About 6 weeks ago i couldn't see it appearing in my list of visitors, but I noticed that I was still getting traffic from google and that the cache was still being updated. Then on 8th december even the cahce stopped being updated. I had a feeling it could be a Blogplay widget that I installed on 8th dec so i removed this and within 10 minutes googlebot was visiting my page again and it has done another 2 or 3 times since. however I can see that the cahe view has still not changed since 8th Dec. So now it seems googlebot is visiting but not caching my page? Do you have any ideas if I have a problem here? Or what I can do about it? When considering your response please bear in mind I'm not knowledgable about SEO terminology :) Thanks for your thoughts.

    • http://imtips.co Shabbir Bhimani

      There is nothing wrong that you have done and its just that your homepage is not updated in cache but your 14th December countdown post is already in Google's Cache and also I did not see anything wrong in your site.

      • adorigraphics

        Oh, when I look at the cache my most recent cached post says 8th Dec. You must be finding it a different way to me (I'm typing my site name into google and then clicking on cache). I will take your word for it that nothing is wrong since I don't have a clue. Thanks so much for taking the time to respond and for the reassurance.

        • http://imtips.co Shabbir Bhimani

          No you are absolutely right and your homepage cache was of 8th December only but there are other pages of your site as well and apply the same method on pages and posts of 14th December and you will see that its also cached which means there is nothing wrong in caching of posts and home page cache is not updated which is perfectly normal

  • ODOnline

    Thank you very much for this wonderfull article

    I was wondering why my website isn't geting craled by Google.
    The most of the people who I asked told me that it was because it won't crawl the pages which are in my language, but now reading your article I know that it is because my site isn't getting enough in-links and also it's new, 3 months since I bought the domain and 1 month since I written some content

    Keep up the good posts!