CS 6242 - Digital Libraries

Last updated: Fri Jul 15 23:54:50 CST 2011 - These pages are likely to be out of date as soon as they are published. Please share your resources that you find on IVLE, and let us know whether we can repost the information here.

Digital Library Research Resources

Remember, good research always teaches other researchers something new.

I do not expect you to write any code from scratch. In fact, if you have an account on sf3/sunfire, you can access a host of related software that I use in my research, in the NLP/IR software repository. Feel free to suggest to me other resources that you feel would be useful to have installed and available to the class. Also, please contact me if your quota of disk space is not sufficient for you to do the scale of research that you need.

A few highlighted resources in the NLP/IR software repository that can help you do research for this course are:

NUS SMS corpus - a corpus of SMS messages collected from students here, through various means. Use the current corpus; find it by web searching for for it.
NUS / Excite query logs - one day logs of (the old) Excite search engine, and ten months of queries for the NUS LINC system (parsed and grouped by sessions).
WT10G - a 10 gigabyte collection of web documents, used for standard experiments.
WEKA / SVMlight / LibSVM / Boostexter / etc. - easy to use machine learning utilities. Probably the easiest to use is Boostexter; and the most complete one is WEKA. WEKA includes a number of different machine learners so that one can do a comparative analysis of different machine learning algorithms on your data.
Open directory project RDF dump - the data structure and content of the ODP, a Yahoo! like repository of categorized websites.
WebBase statistics - document frequencies for a large portion of tokens that appear on the web. Can be used for TF*IDF calculations among other things.

Aside from these tools, if you are doing any programming based project that are requires natural language processing, you may want to consider either Java or Python. Everyone already knows Java, but Python perhaps requires a bit more introduction.

Python is supported by the Natural Language Processing Toolkit (NLTK), which is a great resource for using natural languages. It deserves a course on its own, but you can rapidly get the basics (or ask Min in class or on the forum). Warning: NLTK is great, but not compliant with Python version 3.x, so if you are using Python, try out 2.6.6, known to play well with NLTK.

Python is used as a first language in some CS faculties. John M Zelle from Wartburg College has created a highly-rated (accordingly to Amazon, at least) textbook teaching Programming and simple Data Structures using Python, Python Programming: An introduction to Computer Science, 1st edition. (Second edition covers Py 3, which we are not using. Has useful set of slides and code samples.

Students: if you find other relevant links that you feel other students would benefit from, please post it to the general discussion forum AND email the instruction staff.

Min-Yen Kan <kanmy@comp.nus.edu.sg> Wed Dec 8 13:48:45 SGT 2010 | Version: 1.0 | Last modified: Fri Jul 15 23:54:53 2011

CS 6242 - Digital Libraries

Menu

Digital Library Research Resources