Media Cloud

We’re happy to announce that we’re releasing the first version of the Media Cloud code.

This code does three things:

• Runs a web app that allows you to manage a set of media sources and their feeds.
• Periodically crawls the feeds setup within the web app and downloads any new stories found within the downloaded feeds.
• Extracts the substantive text from the downloaded story content (minus the ads, navigation, comments, etc) and associates a set of tags with each story based on that extracted text.

The system is written perl on top of postgres and uses the catalyst web application framework for the web application.

We’ve been running the code for almost a year now in production, but we’re publishing this as an alpha release because we have not extensively documented the installation or use of the system.