P
Pete Skomoroch
datawrangling.com · Personal blog · on HN since 2007
Pete Skomoroch is a data scientist who writes about datasets, machine learning, and large-scale data processing with tools like Hadoop and AWS.
What HN saysCommenters share his lists of public datasets and free video courses in math, science, and engineering, plus a post on FlightCaster's flight predictions.
20HN people shared it
14linked it in comments
12stories submitted
242story points
12 years agolast shared
2007people per year2026
Most-shared posts
- 1Some Datasets Available on the Web8 people · 47 points · first shared 2008 · 6 comments on HN
- 2How FlightCaster Squeezes Predictions from Flight Data4 people · 93 points · first shared 2009 · 16 comments on HN
- 3Hidden video courses in math science and engineering3 people · first shared 2008
- 4List of Datasets on the Web2 people · 31 points · first shared 2008 · 4 comments on HN
- 5Hidden Video Courses in Math, Science, and Engineering1 people · 37 points · first shared 2008 · 6 comments on HN
- 6Python Montage Code for Displaying Arrays1 people · 14 points · first shared 2008 · discussion on HN
- 7Amazon Web Services Public Datasets1 people · 6 points · first shared 2008 · 1 comments on HN
- 8Amazon Elastic MapReduce: A Web Service API for Hadoop1 people · 5 points · first shared 2009 · 1 comments on HN
- 9MPI Cluster with Python and Amazon EC21 people · 4 points · first shared 2008 · discussion on HN
- 10Wikipedia Page Traffic Statistics Dataset1 people · 3 points · first shared 2009 · discussion on HN
- 11On-Demand MPI Cluster with Python and EC21 people · 2 points · first shared 2008 · discussion on HN
- 12Google paper on parallel em algorithm using mapreduce1 people · first shared 2007
- 13Python montage code for displaying arrays1 people · first shared 2008
- 14Homepage1 people · first shared 2010
What HN people wrote when they linked it
grrrr · 2010-05-04
The book by Manning (freely available online) has already been recommended. I would start with this.
In addition there are a wealth of online video lectures that may inspire you:
http://www.datawrangling.com/hidden-video-courses-in-math-science-and-…
and
http://videolectures.net/mlss04_hofmann_irtm/
and
http://videolectures.net/Top/Computer_Science/
In so far as search engines go it's certainly worth playing around with Lucene. It's well implemented and you'll learn a lot of what really matters when it comes to indexing and retrieval.
For the text processing (classification, data extraction) side It may also be worth brushing up on your stats (a good excuse to learn R) and checking out Mahout http://lucene.apache.org/mahout/
keefe · 2009-06-19
I think there are some interesting lectures here http://www.datawrangling.com/hidden-video-courses-in-math-science-and-… and I occasionally find something insightful here http://demonstrations.wolfram.com/ but I'm in a similar position as you, unfortunately...
dood · 2007-08-12
It depends on the context you are using it in! For social news (e.g. reddit et al.), this sort of thing might be a great help at providing personal recommendations.
Here is a fun recent paper from google, about 'Scalable Online Collaborative Filtering' for their personalised news service [http://www2007.org/paper570.php], using MapReduce to run Expectation Maximization. via [http://www.datawrangling.com/google-paper-on-parallel-em-algorithm-usi…
diegogomes · 2011-10-17
Another cool related list can be found here:
http://www.datawrangling.com/some-datasets-available-on-the-web
clyfe · 2011-09-23
jimbokun · 2010-11-08
Otherwise known as "wrap the crap."
http://www.datawrangling.com/how-flightcaster-squeezes-predictions-fro…
"Building layer upon layer of abstraction is a big key. On the jvm, you have to do this, it is the path around the verbosity of Java and the vast abyss of poorly done APIs. You just keep searching until you finally find the folks who have built a sane, high level API on top of the thing you want to use - then you wrap it in a high level language like Clojure. The technical term for this is 'wrap the crap.'"
helwr · 2010-07-07
pronoiac · 2010-01-01
From http://www.infochimps.org/ -
* http://www.datawrangling.com/
* http://flowingdata.com/
* http://numbrary.com/
* http://swivel.com/
* http://theinfo.org/
* http://aws.amazon.com/publicdatasets/
http://www.opendatacommons.org/
http://delicious.com/pskomoroch/redistributable+dataset
http://www.readwriteweb.com/archives/where_to_find_open_data_on_the.ph…
physcab · 2009-08-15
uc irvine machine learning repository. http://archive.ics.uci.edu/ml/
it's where the netflix dataset is headed. you'll find many datasets there that have been used in scientific papers as well, so you can benchmark whatever tests you want to run.
the data wrangling blog also has a pretty good list of datasets: http://www.datawrangling.com/some-datasets-available-on-the-web
pskomoroch · 2008-11-13
Numpy/Scipy syntax is very close to matlab, but python is a lot more powerful. I port matlab code over to python pretty frequently.
A toy example is here along links to other comparisons:
http://www.datawrangling.com/python-montage-code-for-displaying-arrays
nreece · 2008-04-10
A Dataset Catalog: http://www.datawrangling.com/some-datasets-available-on-the-web.html
sebg · 2014-11-21
Kaggle => http://www.kaggle.com/
List of data sets => http://www.datawrangling.com/some-datasets-available-on-the-web
Another list of data sets => http://blog.mortardata.com/post/67652898761/6-dataset-lists-curated-by…
List of data sets on Quora => http://www.quora.com/Where-can-I-find-large-datasets-open-to-the-publi…
Sub-reddit for data sets => http://www.reddit.com/r/datasets
seancron · 2011-02-01
Here's some more links to data sets:
http://radar.oreilly.com/2010/03/open-data-pointers.html
http://www.datawrangling.com/some-datasets-available-on-the-web
http://del.icio.us/pskomoroch/dataset
http://infochimps.com/collections/datamob (and the other collections on the site)
http://www.data.gov/
vijayr · 2009-09-24
If you are interested in data, here are some sites to get them from
http://theinfo.org/get/data
http://infochimps.org
http://developer.amazonwebservices.com/connect/kbcategory.jspa?categor…
http://ckan.net
EDIT: Comprehensive list here
http://www.datawrangling.com/some-datasets-available-on-the-web