Friday, August 01, 2014
CategoriesMinimize
All Entries
  

ArchivesMinimize
  

ArticlesMinimize
 

Trawling the Deep Web - The XMod Way

Created 5 years 272 days ago
by Administrator Account

I am the fatgeorge.co.uk administrator.
Like: 0 Dislike: 0

  • Currently 0.00/5 Stars.


Tags:
Categories:categoryXModcategoryTips & TrickscategoryHow To
Views: 4205
Print

Lately I have noticed several forum posts and articles discussing the ‘Deep’ or ‘Hidden’ web both in relation to DNN and XMod.

I had never really thought about this before but when pointed out to you things become obvious if your XMod records are not directly linked then how do Google or other search engines index them?

The short answer is they don’t; if your content is dynamic it is almost certain that it is not being indexed, the DNNProfessor article provides links to some modules that help to create sitemaps for DNN.  However many of the processes for generating sitemaps have the same problem as the search engine crawlers, they don’t know about your hidden data.  You could hand-crank a sitemap but where is the fun in that!  Plus if you have a large amount of rapidly changing data on your site your time would be better spent painting the Forth Bridge.

I decided to investigate what was possible via Google’s Webmaster tools, here I read up on the use of sitemaps in particular Creating a Sitemap; here I learnt that Google accepts RSS 2.0 feeds as a Sitemap!  Further investigation revealed that Microsoft Live Search and Yahoo also accept RSS 2.0 feeds.

Now it gets interesting, for those of you that don’t know fatgeorge produce Syndication for XMod, an add-on to create RSS 2.0 feeds from your XMod data.

I already had fatgeorge.co.uk setup in my Google Webmaster tools account and had already submitted the DNN generated sitemap, so I decided to try adding an existing RSS feed as a sitemap.

The Ask The Xperts XMod application has two RSS feeds, one for all records and one for un-answered questions; I added the first feed to Google which was a very simple task, basically just enter the url of the RSS xml file on your site.RSS Sitemap

 

Google read the feed fairly quickly but did not immediately index the url’s, initially I kept checking daily to see if the feed’s url's had been indexed, then every other day, then whenever I remembered.

Today, I checked and Ta Daaaa the feed has been indexed.

 

 

 

 

 

 

 

 

 

 

After I initially submitted the feed to Google I did a very un-scientific test by searching for one of the ATX questions which concerned ‘uploading 2 files with XMod’, it did return some results; some from DNNDev about XMod and some others about DNN but none from fatgeorge.

After I saw that the feed had been indexed I tried searching again.

Google search results

As I said this is very un-scientific, my next test is to add a new question to ATX and see how long it takes Google to download the updated feed and index it.

 

The process to submit your RSS feeds as sitemaps on Live Search and Yahoo is very similar and just as easy, but only Google give the detailed breakdown of the feed and it’s status.

 

image

 

More about the Deep or Invisible web:


image24.jpg 

No Comments


You need to register to comment.
  

Copyright (c) 2014 fatgeorge online solutions