How search engines index blogs
I learned something interesting about how Harvard’s search engine (powered by Inktomi) indexes blogs today. A few days ago, I traced a referral Google made to my site. Google only seems to index the first page of this blog. Beth Potier, one of my coworkers, told me that she found my blog while she was doing some research on the USA Patriot Act using Harvard’s search engine. When I pulled up my blog’s homepage, she told me that it looked nothing like that. What she saw was just the individual post. It appears that Harvard’s search engine goes deeper into the blogs than just the front page and uses better URLs. Getting a search engine to use the permalinks is still an issue, though.
