Affiliate Marketing
Forum Search

Reply
 
LinkBack Thread Tools Display Modes

  #1 (permalink)  
Old 24-09-07
Registered User
 
Join Date: Apr 2004
Location: UK
Posts: 380
Thanks: 3
Thanked 0 Times in 0 Posts
Matt Seigneur is an unknown quantity at this point
  robots.txt file - a must?

just wondering if the robots.txt file is a must in SEO?

Cheers
Matt
__________________
Complete affiliate web design service including 100% unique content and SEO - pm for more details
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #2 (permalink)  
Old 24-09-07
Registered User
 
Join Date: Sep 2007
Posts: 582
Thanks: 0
Thanked 0 Times in 0 Posts
cyates is an unknown quantity at this point
  Re: robots.txt file - a must?

nope

West Midlands Finance - Birmingham Secured Loans, Coventry Commercial Mortgages, Wolverhampton IVA doesnt have 1 .. only been designed & uploaded 2 days already all of teh pages are indexed except the new ones i just added today
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #3 (permalink)  
Old 24-09-07
Registered User
 
Join Date: Nov 2004
Location: Wirral, Merseyside
Posts: 1,001
Thanks: 3
Thanked 4 Times in 2 Posts
gadget is an unknown quantity at this point
  Re: robots.txt file - a must?

Well that blew my understanding!

cyates - do you have any sitemaps?
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #4 (permalink)  
Old 24-09-07
morleymouse's Avatar
Super Member
 
Join Date: Aug 2003
Location: Costa Del Sheffield
Posts: 2,708
Thanks: 0
Thanked 1 Time in 1 Post
morleymouse is an unknown quantity at this point
  Re: robots.txt file - a must?

id always add one just to keep it out of the error logs, but thats just personal preference
__________________
Dan Morley
Alpharooms.com
daniel at alpharooms dot com - Hotels, Flights, Airport Transfers, Care Hire + More! sign up
My Blog | Cheap Holidays
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #5 (permalink)  
Old 24-09-07
Registered User
 
Join Date: Apr 2004
Location: UK
Posts: 380
Thanks: 3
Thanked 0 Times in 0 Posts
Matt Seigneur is an unknown quantity at this point
  Re: robots.txt file - a must?

Quote:
Originally Posted by morleymouse View Post
id always add one just to keep it out of the error logs, but thats just personal preference
yeah true, thats why i was asking actually, what should be on the robots.txt file?
__________________
Complete affiliate web design service including 100% unique content and SEO - pm for more details
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #6 (permalink)  
Old 24-09-07
morleymouse's Avatar
Super Member
 
Join Date: Aug 2003
Location: Costa Del Sheffield
Posts: 2,708
Thanks: 0
Thanked 1 Time in 1 Post
morleymouse is an unknown quantity at this point
  Re: robots.txt file - a must?

if your not bothered just stick a blank text file there
__________________
Dan Morley
Alpharooms.com
daniel at alpharooms dot com - Hotels, Flights, Airport Transfers, Care Hire + More! sign up
My Blog | Cheap Holidays
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #7 (permalink)  
Old 24-09-07
Registered User
 
Join Date: Sep 2007
Posts: 582
Thanks: 0
Thanked 0 Times in 0 Posts
cyates is an unknown quantity at this point
  Re: robots.txt file - a must?

nope no sitemaps at all
the key is interlinking as many pages as possible
i do have a very nice little trick for large sites though
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #8 (permalink)  
Old 25-09-07
Registered User
 
Join Date: Nov 2004
Location: Wirral, Merseyside
Posts: 1,001
Thanks: 3
Thanked 4 Times in 2 Posts
gadget is an unknown quantity at this point
  Re: robots.txt file - a must?

Quote:
Originally Posted by cyates View Post
nope no sitemaps at all
the key is interlinking as many pages as possible
i do have a very nice little trick for large sites though
Are you going to share it or just tease us?

I've always been careful not to interlink too much across my sites to avoid any triangulation penalties, especially if they're on the same IP blocks. What are your thoughts on this?
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #9 (permalink)  
Old 25-09-07
Registered User
 
Join Date: Sep 2007
Posts: 582
Thanks: 0
Thanked 0 Times in 0 Posts
cyates is an unknown quantity at this point
  Re: robots.txt file - a must?

i dont interlink the sites.. just the pages interlinking with pages on the same domain

also i'll copy a pm i just sent to stuart in my next post
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #10 (permalink)  
Old 25-09-07
Registered User
 
Join Date: Sep 2007
Posts: 582
Thanks: 0
Thanked 0 Times in 0 Posts
cyates is an unknown quantity at this point
  Re: robots.txt file - a must?

ok lol
take a look at West Midlands Finance - Birmingham Secured Loans, Coventry Commercial Mortgages, Wolverhampton IVA

all pages have the menu on the left in them
also a good trick is using CSS create anchor text links with the same font style and size as normal text on the page so that words can be hyperlinks without drawing the visitors attention to it

so for example

Aroma restaurants are modern “eat as much as you like for a fixed price” Oriental buffet restaurants. All our restaurants are designed along clean, modern lines where diners can enjoy oriental cuisine in a stylish atmosphere at an affordable price. Starters, main courses, tepanyaki and desserts are included in the fixed price. Diners can choose from as many as 30 lunchtime or 60 dinner dishes offering culinary delights from the Far East - you will be spoilt for choice

loads of those words would be hyperlinked to either the same page or other pages that contain information about them

another great way to get all teh pages indexed really quickly
is to create an RSS feed

all you need to do is edit the following
HTML Code:
<?xml version="1.0"?> 
<rss version="2.0">
<channel>
<item>
<title>PAGE TITLE</title>
<description>Description about the page you are linking to</description>
<link>http://www.mydomain.co.uk/page1.php</link>
</item>
<item>
<title>2ND PAGE</title>
<description>Description about the page you are linking to</description>
<link>http://www.mydomain.co.uk/page2.php</link>
</item>
</channel>
</rss>
then save as link.xml
and PING your website Blog and ping | Pingoat
remember to include the link to your rss feed ( your XML link map )

Chris
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #11 (permalink)  
Old 25-09-07
Registered User
 
Join Date: Nov 2004
Location: Wirral, Merseyside
Posts: 1,001
Thanks: 3
Thanked 4 Times in 2 Posts
gadget is an unknown quantity at this point
  Re: robots.txt file - a must?

Nice post Chris.

.. hi-ho, it's off to work I go ..
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
Reply

Bookmarks
  • Submit Thread to Digg