Tools for text based research

Tools for text based research

Crawling the internet: There are plenty of toolkits for doing this, like Scrapy for Python , crawler4j and Heritrix for Java, or WWW::Robot for Perl. For extracting the actual content from web pages, have a look at boilerpipe post in my blog.

1. http://scrapy.org/
   http://crawler.archive.org/
   http://code.google.com/p/crawler4j/
   https://metacpan.org/module/WWW::Robot
   http://code.google.com/p/boilerpipe/
2. often you can use collaborative filtering instead of content-based approaches. But if you want to have good coverage, especially in the long tail, there will be no way around analyzing the text. One thing to look at is topic modelling, e.g. LDA. Several LDA approaches are implemented in Mallet, Apache Mahout, and Vowpal Wabbit. For indexing, search, and text processing, have a look at Lucene. It is an awesome, mature piece of software.
   http://mallet.cs.umass.edu/
   http://mahout.apache.org/
   http://hunch.net/~vw/
   http://lucene.apache.org/
3. Other then Apache Mahout which also contains things like LDA (see above), clustering, and text processing, there are also other toolkits available if you want to focus on collaborative filtering: LensKit, which is also implemented in Java, and MyMediaLite 
   http://lenskit.grouplens.org/
   http://ismll.de/mymedialite
   https://github.com/jcnewell/MyMediaLiteJava