robots.txt file - a must?

[Matt Seigneur]

Tweet

just wondering if the robots.txt file is a must in SEO?

Cheers
Matt

[cyates]

nope

West Midlands Finance - Birmingham Secured Loans, Coventry Commercial Mortgages, Wolverhampton IVA doesnt have 1 .. only been designed & uploaded 2 days already all of teh pages are indexed except the new ones i just added today

[gadget]

Well that blew my understanding!

cyates - do you have any sitemaps?

[morleymouse]

id always add one just to keep it out of the error logs, but thats just personal preference

[Matt Seigneur]

id always add one just to keep it out of the error logs, but thats just personal preference

yeah true, thats why i was asking actually, what should be on the robots.txt file?

[morleymouse]

if your not bothered just stick a blank text file there

[cyates]

nope no sitemaps at all
the key is interlinking as many pages as possible
i do have a very nice little trick for large sites though

[gadget]

nope no sitemaps at all
the key is interlinking as many pages as possible
i do have a very nice little trick for large sites though

Are you going to share it or just tease us?

I've always been careful not to interlink too much across my sites to avoid any triangulation penalties, especially if they're on the same IP blocks. What are your thoughts on this?

[cyates]

i dont interlink the sites.. just the pages interlinking with pages on the same domain

also i'll copy a pm i just sent to stuart in my next post

[cyates]

ok lol
take a look at West Midlands Finance - Birmingham Secured Loans, Coventry Commercial Mortgages, Wolverhampton IVA

all pages have the menu on the left in them
also a good trick is using CSS create anchor text links with the same font style and size as normal text on the page so that words can be hyperlinks without drawing the visitors attention to it

so for example

Aroma restaurants are modern “eat as much as you like for a fixed price” Oriental buffet restaurants. All our restaurants are designed along clean, modern lines where diners can enjoy oriental cuisine in a stylish atmosphere at an affordable price. Starters, main courses, tepanyaki and desserts are included in the fixed price. Diners can choose from as many as 30 lunchtime or 60 dinner dishes offering culinary delights from the Far East - you will be spoilt for choice

loads of those words would be hyperlinked to either the same page or other pages that contain information about them

another great way to get all teh pages indexed really quickly
is to create an RSS feed

all you need to do is edit the following

<?xml version="1.0"?> 
<rss version="2.0">
<channel>
<item>
<title>PAGE TITLE</title>
<description>Description about the page you are linking to</description>
<link>http://www.mydomain.co.uk/page1.php</link>
</item>
<item>
<title>2ND PAGE</title>
<description>Description about the page you are linking to</description>
<link>http://www.mydomain.co.uk/page2.php</link>
</item>
</channel>
</rss>

then save as link.xml
and PING your website Blog and ping | Pingoat
remember to include the link to your rss feed ( your XML link map )

Chris

[gadget]

Nice post Chris.

.. hi-ho, it's off to work I go ..