Comments on: Thought du jour http://blogs.law.harvard.edu/doc/2008/05/29/thought-du-jour/ Same old blog, brand new place Wed, 11 Nov 2009 02:29:04 -0500 http://wordpress.org/?v=2.8.4 hourly 1 By: mikepk http://blogs.law.harvard.edu/doc/2008/05/29/thought-du-jour/comment-page-1/#comment-55287 mikepk Fri, 30 May 2008 22:35:23 +0000 http://blogs.law.harvard.edu/doc/2008/05/29/thought-du-jour/#comment-55287 We actually considered open sourcing a version of our technology (grazr) for doing Twitter-like things. We serve several million feeds per month based on reading lists (opml) as well as "streams" (merged river-of-news like feeds) that we do with sub second response times (when properly tuned, we're in the middle of some mods right now slowing it down). Not to toot our own horn, but we do all of that with the contents of the feeds (a lot more than 140 characters), so the amounts of data we handle is at least an order of magnitude higher than Twitter (according to my back of the envelope calculations). So yes, it's clearly a solvable problem. We decided not to pursue this path though because we assumed Twitter would get it's act together and that its userbase and general user loyalty would allow it to weather this current storm. That may have been a faulty assumption, I'm not so sure it's going to survive after seeing how badly it's been falling over lately. We actually considered open sourcing a version of our technology (grazr) for doing Twitter-like things. We serve several million feeds per month based on reading lists (opml) as well as “streams” (merged river-of-news like feeds) that we do with sub second response times (when properly tuned, we’re in the middle of some mods right now slowing it down).

Not to toot our own horn, but we do all of that with the contents of the feeds (a lot more than 140 characters), so the amounts of data we handle is at least an order of magnitude higher than Twitter (according to my back of the envelope calculations). So yes, it’s clearly a solvable problem.

We decided not to pursue this path though because we assumed Twitter would get it’s act together and that its userbase and general user loyalty would allow it to weather this current storm. That may have been a faulty assumption, I’m not so sure it’s going to survive after seeing how badly it’s been falling over lately.

]]> By: Mike Warot http://blogs.law.harvard.edu/doc/2008/05/29/thought-du-jour/comment-page-1/#comment-55262 Mike Warot Fri, 30 May 2008 19:15:14 +0000 http://blogs.law.harvard.edu/doc/2008/05/29/thought-du-jour/#comment-55262 It continues to amaze me that a flow of about 10 tweets/second is too much for twitter to handle. It's obvious that they need a complete redesign of their architecture from one based on an overworked RDBMS to one based on message queues and a small cluster of UI engines on the outskirts. I think it would be a cool idea to have someone sponsor a contest to see who could come up with a twitter core replacement. You'd have to be able to sustain traffic rates of at least 10 tweets/second at the core just to be able to enter the contest. I'd be willing to bet that someone could figure out how to get at least 1000 times flow as a core router, with the UI boxes handling at least 100 simultaneous users. It's not rocket science, but it does need computer science. --Mike-- It continues to amaze me that a flow of about 10 tweets/second is too much for twitter to handle. It’s obvious that they need a complete redesign of their architecture from one based on an overworked RDBMS to one based on message queues and a small cluster of UI engines on the outskirts.
I think it would be a cool idea to have someone sponsor a contest to see who could come up with a twitter core replacement. You’d have to be able to sustain traffic rates of at least 10 tweets/second at the core just to be able to enter the contest. I’d be willing to bet that someone could figure out how to get at least 1000 times flow as a core router, with the UI boxes handling at least 100 simultaneous users.
It’s not rocket science, but it does need computer science.
–Mike–

]]> By: tk http://blogs.law.harvard.edu/doc/2008/05/29/thought-du-jour/comment-page-1/#comment-55251 tk Fri, 30 May 2008 17:56:24 +0000 http://blogs.law.harvard.edu/doc/2008/05/29/thought-du-jour/#comment-55251 COMPETITION IS THE BEST INCENTIVE, who will set up shop and enter the fray? COMPETITION IS THE BEST INCENTIVE, who will set up shop and enter the fray?

]]> By: david cushman http://blogs.law.harvard.edu/doc/2008/05/29/thought-du-jour/comment-page-1/#comment-55168 david cushman Fri, 30 May 2008 08:12:56 +0000 http://blogs.law.harvard.edu/doc/2008/05/29/thought-du-jour/#comment-55168 What we need is a way to share our metadata without silos, so we can better connect around shared, right now, purpose. IMHO, of course. What we need is a way to share our metadata without silos, so we can better connect around shared, right now, purpose. IMHO, of course.

]]> By: Dave Winer http://blogs.law.harvard.edu/doc/2008/05/29/thought-du-jour/comment-page-1/#comment-55142 Dave Winer Fri, 30 May 2008 05:45:33 +0000 http://blogs.law.harvard.edu/doc/2008/05/29/thought-du-jour/#comment-55142 An open source client will enable multiple back-ends, transparent for the users. An open source client will enable multiple back-ends, transparent for the users.

]]>