Aggregator woes
July 30th, 2003
The past few days I have noticed very strange referrals from various search engines in my referer stats so I decided to investigate. I was relieved to find that some of the news feeds in my aggregator were the source of all of the strange search engine hits.
Daypop, which I added over the weekend just out of curiosity, was the biggest troublemaker. People searching Google for any of the Daypop Top 40 items ended up hitting the page for my aggregator because, it seems, my aggregator is being indexed. Although I sort of liked the illusion of having readers, I did not like the creepy and sometimes obscene combinations of search terms in my referer log. Plus, my inner librarian was not comfortable with (unintentionally) luring people to my page with information that was not there.
So, I removed the troublesome feeds, and now my blog will return to desperate obscurity, excepting the people searching for Latin dictionaries and the Lindisfarne Gospels, Miss Piggy and Mister Rogers, and–my personal favorite–library party blogs (and all of the variations).
Has anyone else noticed weird search engine hits in their referer logs because of aggregator indexing? And is it particularly useful for search engines to index aggregators? Or is this a possible glitch in the improvements to Google indexing that Jessica (aka J of J’s Scratchpad) observed? Just wondering.
Entry Filed under: Administrative Matters
2 Comments Add your own
Leave a Comment
You must be logged in to post a comment.
Trackback this post | Subscribe to the comments via RSS Feed
1. Desultor | July 30th, 2003 at 5:04 pm
It’s weird and confusing! I think the issue here is that Google doesn’t really make any distinction between your aggregator content and any other content - they just have a bunch of web-robots which spider around and look at any web pages they can find, whether they’re human-generated or not. They might be trying to ignore aggregators, but it’s hard to teach a dumb robot how to recognize one!
The silver lining for this cloud is that these misdirections are mainly transient - as your aggregator moves on to other articles, Google will not catch the search terms next time. In a week or two, the offending search terms will have fallen off the bottom of most people’s aggregators - and since the original content you aggregated will probably still be around, Google will send people to it instead of your aggregator.
I find that people’s search results matter a lot to me too. For instance, my blog had a post where I talked about the etymology of the word “juke” - but not the same juke as in “jukebox”. I get queries for “etymology juke” and feel guilty - I’m pretty sure most people have the other “juke” in mind, and I feel like I should put that one’s etymology up too!
2. Vernica | July 31st, 2003 at 3:27 pm
Aha…thanks, Jesse, your comment solved my problem. I just discovered on the advanced preferences page that there is an option to block selected weblog pages from crawlers and search engines. The referer log was already selected, and I added my aggregator. I never noticed that option before. I wonder if that was added recently or if I just missed it? At any rate, that should solve the weird search engine indexing problem.
It is also comforting to know that I’m not the only person thinking about other people’s search queries :-).