David MacKay

I've recently discovered this excellent lecture series by David Mackay available on YouTube: https://youtu.be/y5VdtQSqiAI He also wrote the accompanying text book which is available for free download: http://www.inference.phy.cam.ac.uk/itprnn/book.pdf I was really impressed by these lectures, and was dismayed to learn that he died from cancer a couple of years ago.

Great idea! This book is excellent, and though the principles are timeless, it would be a shame if the data became outdated. Sadly the source code for the book doesn't appear to be published. This is unlike David Mackay's otherbook, "Information Theory, Inference & Learning Algorithms", does have the LaTeX source published (see http://www.inference.phy.cam.ac.uk/mackay/itila/book.html ), though not in an open source license.

+1 for Information Theory, Inference, and Learning Algorithms. A fantastic book by the late David C Mackay. Part V has an interesting presentation of neural networks, but really this book is about information theory and Bayesian probability. http://www.inference.phy.cam.ac.uk/itila/

Sanjoy Mahajan [0], who is the author of perhaps the two best books on Fermi estimation (freely available online, too!), has linked on his website a really fascinating account of an experiment to reform mathematical K-12 education from 70+ years ago [1]. [0] http://web.mit.edu/sanjoy/www/ [1] http://www.inference.phy.cam.ac.uk/sanjoy/benezet/

In terms of gaining a solid understanding of the math and theory underlying most of modern machine learning/AI, the best resource is definitely the book "Information theory, inference and learning algorithms" by David MacKay. The author is such a stand-up guy, he's made it available for free: http://www.inference.phy.cam.ac.uk/itprnn/book.pdf

some useful books available as a PDF: David Barber "Bayesian Reasoning and Machine Learning" http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=Brml.HomeP… -- Thorough and systematic, accompanied by a Matlab library Marc Steyvers "Computational Statistics with Matlab" http://psiexp.ss.uci.edu/research/teachingP205C/205C.pdf -- Somewhat unfinished but very good hands-on guide for a total novice David Draper "Bayesian Modeling, Inference and Prediction" https://users.soe.ucsc.edu/~draper/draper-BMIP-dec2005.pdf -- Quite good if you're already familiar with a field, might be slightly daunting for a beginner and the classic, David MacKay's "Information Theory, Inference, and Learning Algorithms" http://www.inference.phy.cam.ac.uk/itprnn/book.html

I wonder how well the hardware would work with an input system like Dasher. Dasher was experiment in statistical inference and tries to be an accessibility tool as well. http://www.inference.phy.cam.ac.uk/dasher/. I found it really fun to play around with and could get decent speeds but not great. It's true about eye tracking being prohibitively high. When we order an eye tracker for the lab, it ended up costing us £10K.

I will check those out, thanks. This is indeed a fascinating topic. I guess every scientist wants to understand learning. The connection comes up in david mckay's Information Theory book too, a reading I definitively recommend, although I haven't been through it properly myself. http://www.inference.phy.cam.ac.uk/itprnn/book.pdf

You could look at the appendixes in texts by Bishop, MacKay, Barber and Rasmussen/Williams to see what they expect (and they expect a pretty thorough understanding. The last 3 are freely available content http://www.inference.phy.cam.ac.uk/itila/book.html http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/090310.pdf http://www.gaussianprocess.org/gpml/chapters/RW.pdf Also, an excellent list of ML resources: http://www.reddit.com/r/MachineLearning/comments/1jeawf/machine_learni…

When alternative text input comes up I always feel compelled to share Dasher[1]. Dasher is designed with accessible computing in mind, and can be driven using a wide variety of input method. The project appears to be abandoned - so no leap support - though the leap would be an excellent fit for the interface, minus the gorilla arm problem. [1] http://www.inference.phy.cam.ac.uk/dasher/

I would add to that two classics with free PDFs available from the authors' websites: 1. David Barber: Bayesian Reasoning and Machine Learning 2. David MacKay: Information Theory, Inference, and Learning Algorithms [1] http://www.cs.ucl.ac.uk/staff/d.barber/brml/ [2] http://www.inference.phy.cam.ac.uk/mackay/itila/ Both require a willingness to immerse in the mathematics. But the maths isn't hard, even for someone like me who hasn't formally studied it since I was 15.

As a rather short (9 pages) introduction to Bayesian inference, I remember greatly appreciating an extract from the book "Information Theory, Inference, and Learning Algorithms" by David MacKay. The relevant part is on PDF pages 469 to 477 here: <http://www.inference.phy.cam.ac.uk/itprnn/book.pdf> It explains how Bayesian inference works (which is intuitive and not strange at all) and gives examples of absurdities that comes upon you when you don't do it this way. Highlights: • "Let me through, I’m a Bayesian" (When analysing the effectivness of a vaccine) • "I have no problem with the idea that there is only one answer to a well-posed problem" (In response to sampling theorists wide selection of ad hoc procedures)

There looks to be lots of solar both in urban and rural locations in the UK, thanks to generous grants/assured tariffs. Prof David MacKay talks about equalising supply from wind & solar with pumped hydro and electric vehicles in his excellent book, page 190 onwards: http://www.inference.phy.cam.ac.uk/sustainable/book/tex/sewtha.pdf

This is addressed in the "Renewable Energy - Without the Hot Air" Report http://www.inference.phy.cam.ac.uk/withouthotair/c26/page_187.shtml basically the author says that what you describe is the best option right now but there are only so many places you can put those facilities because they require very specific geography. That being said building lakes and dams is really bad for the river eco system and really expensive to buy land out from under people to flood it.

i'm not going to argue here because http://www.withouthotair.com/ makes the point better than i ever could. see, for example, http://www.inference.phy.cam.ac.uk/withouthotair/c19/page_114.shtml - Don’t be distracted by the myth that “every little helps.” If everyone does a little, we’ll achieve only a little. We must do a lot. What’s required are big changes in demand and in supply. and, of course, we may be talking at cross purposes - there is some middle ground where a large-ish number of small-ish things can help. i am not arguing against maths, but i am saying that our natural assessment of what is significant without actually doing the maths is often misled by absolute values, as here. anyway, if you haven't read that book, i recommend it.

David MacKay's book "Sustainable Energy - without the hot air" goes into a huge amount of detail about plane efficiency http://www.inference.phy.cam.ac.uk/withouthotair/cC/page_269.shtml It's also a great book

By far my favorite book on the subject is conveniently available for free on the internet! "Information Theory, Inference and Learning Algorithms" by David MacKay is wonderfully written, well paced and comprehensive. If you like the book, you should purchase a copy, David is a great guy. http://www.inference.phy.cam.ac.uk/mackay/itila/book.html Videolectures has some very good videos as well. Zoubin Gharamani has a pretty solid lecture on Bayesian learning at http://videolectures.net/mlss05us_ghahramani_bl/ (he's a great researcher but not the most engaging speaker). Try Christopher Bishop's lecture at http://videolectures.net/mlss09uk_bishop_ibi/ as well, it might be slightly more palatable.

> Thus solar and wind fit in with our natural rhythm quite nicely. It's not that nice, because power output from wind/solar plants is random (depends on things like cloud cover or wind strength) and it causes problems in power grid, where power demand must meet the supply exactly, or bad things will happen. "Without Hot Air"[1] covers this, and other renewable-related issues quite nicely and with real data. I recommend it, it's a good read. It has some good ideas on how to solve power supply/demand problems. On TED2012 there was a talk about a new kind of batteries designed to solve those kind of problems in the power grid; the video from the talk is not yet up, though (but I think it should be soon, at [2]). [1] - http://www.inference.phy.cam.ac.uk/withouthotair/ [2] - http://www.ted.com/talks?lang=en&event=2012&duration=&sort…

My current favorite, Information Theory Inference and Learning Algorithms, available online for free. :) http://www.inference.phy.cam.ac.uk/mackay/itila/

The persistence and the dedication needed in skateboarding—that's what we need to be teaching. No one says to a toddler, 'You have ten weeks to walk, and if you can't, you get an F and you're not allowed to try to walk anymore.' It's absurd, right? But the same thing is true with math and science education. If you want to learn trig or calculus, it's set at such a pace in schools that it guarantees that only the absolutely best students will learn it. Reminded me of the essay "Everyone Should Get an A": http://www.inference.phy.cam.ac.uk/mackay/abstracts/exams.html

Do windmills kill “huge numbers” of birds? Wind farms recently got adverse publicity from Norway, where the wind turbines on Smola, a set of islands off the north-west coast, killed 9 white-tailed eagles in 10 months. I share the concern of BirdLife International for the welfare of rare birds. But I think, as always, it’s important to do the numbers. It’s been estimated that 30 000 birds per year are killed by wind turbines in Denmark, where windmills generate 9% of the electricity. Horror! Ban windmills! We also learn, moreover, that traffic kills one million birds per year in Denmark. Thirty-times-greater horror! Thirty-times-greater incentive to ban cars! And in Britain, 55 million birds per year are killed by cats. --David MacKay, "Sustainable Energy -- without the hot air". http://www.inference.phy.cam.ac.uk/withouthotair/c10/page_63.shtml

I'm reading this now. Things like this remind me of why I love the Internet. So much great knowledge, freely available for everyone. It boggles the imagination sometimes. Also, another book that some HN'ers might find interesting is this one: Information Theory, Inference, and Learning Algorithms: http://www.inference.phy.cam.ac.uk/itprnn/book.html

Nice find! For those looking for a good demo: http://www.inference.phy.cam.ac.uk/dasher/images/newdasher.gif Still... Great for firehose text entry, but doesn't address the sort of text manipulation for which vim/emacs have become so popular.

Nice lists, I often recommend these for people who want an introduction to the field: "Mathematical Statistics and Data Analysis" by John A. Rice "All of Statistics: A Concise Course in Statistics" by Larry Wasserman "Pattern Recognition and Machine Learning" by Christopher M. Bishop "The Elements of Statistical Learning" by T. Hastie et al http://www-stat.stanford.edu/~tibs/ElemStatLearn/ "Information Theory, Inference, and Learning Algorithms", David McKay http://www.inference.phy.cam.ac.uk/itprnn/book.html "Introduction to Information Retrieval" - Manning et al. http://nlp.stanford.edu/IR-book/information-retrieval-book.html "The Algorithm Design Manual, 2nd Edition" - Steven Skiena http://www.algorist.com/

May I also recommend a related book, also a free PDF: http://www.inference.phy.cam.ac.uk/mackay/itila/ "Information theory, inference, and learning algorithms" by David Mackay. This book is not a pure machine learning book but it is a very fun read.

http://www.inference.phy.cam.ac.uk/mackay/itila/Potter.html I haven't read it, but this comparison has convinced me of its worth.

Incidentally, David Mackay's group came up with the Dasher eye-tracking user interface and a bunch of cool machine-learning stuff; and for further HN cred, one of his ex-postdocs is CTO of Songkick. http://www.inference.phy.cam.ac.uk/

Another book of his on a different topic but with same great clarity of explanation is "Information Theory, Inference, and Learning Algorithms". The PDF is on his site at http://www.inference.phy.cam.ac.uk/mackay/itila/book.html.

What am I missing? 1) His general point is that no single energy source will be sufficient. We're going to need a combination of many different sources of power, combined with as much improvement in efficiency as we can muster. He concludes that it is very difficult, but not impossible, and that tradeoffs will be necessary. 2) Covering the electrical needs of the world is an easier task than covering the total energy needs, and even covering the current energy needs will not account for increasing demand by the developing world. Still, this is an approach worth exploring. It's covered in considerable detail in Chapter 25: http://www.inference.phy.cam.ac.uk/withouthotair/c25/page_177.shtml

> New solar cells can sometimes take greater than the life of the product to recoup the energy spent on creating them. If you live in, say, Massachusetts [...] you'll probably never net an energy savings. "False. The energy yield ratio (the ratio of energy delivered by a system over its lifetime, to the energy required to make it) of a roof-mounted, grid-connected solar system in Central Northern Europe is 4, for a system with a lifetime of 20 years (Richards and Watt, 2007); and more than 7 in a sunnier spot such as Australia. (An energy yield ratio bigger than one means that a system is A Good Thing, energy-wise.) Wind turbines with a lifetime of 20 years have an energy yield ratio of 80." source: http://www.inference.phy.cam.ac.uk/withouthotair/c6/page_41.shtml (I assumed that the sun in Central Northern Europe shines with an intensity similar to the intensity in Massachusetts.)