robots.txt is quite useful!
i was worried about duplicate content for the search engines because some of my blog entries are in 3 different places. i wasn\'t quite sure how i was going to easily fix this for the b2e blog that i share with janel.
after i realized that i was dumb not to be using robots.tx, i spent the whole 5 minutes figuring out how to do it. the google robots.txt checker says it works, we\'ll see if the references fade though.
for z425..
User-agent: *
Disallow: /files
Disallow: /its-whats-for-dinner
Disallow: /blog
Disallow: /phpBB2tastic
Disallow: /articles
for projectjanel.org
User-agent: *
Disallow: /b2e_blogs/index.php?blog=2
the patten matching is brain-dead which works well for my situation. from what i understand it literally just checks a url letter for letter until it either doesn\'t match or it runs out of characters to match. so for instance.. if i had a directory on z425 of..
/articles2
it would not be spidered since /articles matches. this works great for me b2e blog shared with janel because it can match out the \'blog=2\' bit. luckily b2e seems reasonably consistant with placing \'blog=2\' as the first argument which makes things a lot easier.
i also noticed that a lot of the search results targetted my table of contents pages. lame. easy to fix though..
i just added an extra meta tag to those pages..
now the big question is will the robots obey?
btw. i\'m sure you are all interested how my click empire is doing..
$5.31
it\'s time to retire to florida! flint michigan style.
