Brendan O'Connor

If anyone is interested, this issue from IBM outlines most parts of Watson in great detail. http://brenocon.com/watson_special_issue/

If you bet on the right platform, code can live for a while. For example: this classic paper on compression http://web.stanford.edu/class/ee398a/handouts/papers/WittenACM87Arithm… was published with a C, Unix-based implementation (I put it here http://brenocon.com/WittenNealCleary-ArithmeticEncodin-cacm-87.shar) which compiles and runs fine (just lots of compiler warnings) on modern Linux and OS X, 18 years later.

When I wrote "In this case it measures the relative reduction in MSE" I meant exactly that. The mean squared error of the baseline model which doesn't include the state as a regressor is 0.25 (it predicts always 0.5 - it's off by 0.5 in every case). The mean squared error of the model which includes the state as a regressor is 0.2475 (it predicts 0.45 or 0.55 depending on the state - in both cases it's off by 0.45 with 55% probability and it's off by 0.55 with 45% probability). The mean squared error is directly related to variance when the predictor is unbiased. The ratio of the sum of squares is the same as the ratio of the mean square errors. Edit: http://brenocon.com/rsquared_is_mse_rescaled.pdf "R2 can be thought of as a rescaling of MSE, comparing it to the variance of the outcome response." https://dabruro.medium.com/you-mention-the-average-squared-error-how-i… "Also it is worth mentioning that R-squared (coeff. of determination) is a rescaled version of MSE such that 100% is perfection and 0% implies the same MSE that you would get by simply always predicting the overall mean of the dataset."

As @vitus points out [1], these numbers are pure, unadulterated garbage. Instead of being data, they are wild extrapolations from this [2] actually useful and relatively correct past data. 2,000 bytes over commodity network in 5 us is 400 MB/s (3.2 Gbps). Far faster than commodity residential speeds. 1,000,000 bytes from memory in 741 ns (ridiculous amount of significant figures) is 1.3 TB/s. 1,000,000 bytes from SSD in 12.245 us (ridiculous amount of significant figures) is 81 GB/s. Disk seek in 1.649384 ms (ridiculous amount of significant figures) means we now have 36,000 RPM HDDs. Read 1,000,000 bytes from HDD in 358.968 us (ridiculous amount of significant figures) is 3 GB/s. Every single one of these numbers is wildly incorrect. This page is unequivocal anti-knowledge. [1] https://news.ycombinator.com/item?id=47196505 [2] https://brenocon.com/dean_perf.html

Discussing performance only makes sense in the context of a particular awk implementation, like TFA is doing as well. If you‘re (stuck) on gawk, try setting LANG=C to prevent Unicode/multi-byte regexp execution, or switch to mawk (which according to [1] is much faster than cpython). [1]: https://brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-and-most-e…

> And performance of awk is crap. [...] I had replaced all the awk scripts in python and everything is a lot faster. My experience points exactly the other way: for data-processing tasks, especially streaming ones, even Gawk is a lot faster than Python (pre-3.11), and apparently I’m not the only one[1]. If you’re not satisfied with Gawk’s performance, though, try Nawk[2] or, even better, Mawk[3]. (And stick to POSIX to ensure your code works in all of them.) [1] https://brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-and-most-e… [2] https://github.com/onetrueawk/awk [3] https://invisible-island.net/mawk/

Its linked in the article: http://brenocon.com/dean_perf.html

Depends on the task...sometimes nawk ("one true awk") is faster than gawk. Mawk is almost always faster than either. An older, but good article on that: https://brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-and-most-e…

See the table in this link for a humorous comparison by Tibshirani: https://brenocon.com/blog/2008/12/statistics-vs-machine-learning-fight…

This is true. I've read all the Watson Special Edition papers numerous times[1]. The original Watson work was very good, and adding neural-network based models as alternative question parsers and answer rankers is a clear way forward. It's a pity IBM decided to use the Watson name for everthing they do in ML now. [1] http://brenocon.com/watson_special_issue/

This is probably the most concise introduction to awk I've read. For more extensions[0] and advanced features like arrays of arrays and array sorting, there's also gawk. And for larger files there's performance driven mawk which can drastically increase processing speed[1]. [0]https://www.gnu.org/software/gawk/manual/html_node/Extension-Samples.h… [1]http://brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-and-most-el…

I'm not sure what the difference between statistics and machine-learning is supposed to be, seriously. The theoretical foundation of any machine learning algorithm is probability theory and statistics. Is machine learning implied to be a more "hands on" approach, to focus more on learning a set of techniques than on learning the theoretical underpinning of such techniques? Is it basically "statistical engineering"? The least charitable interpretation would be that it's just a hipper term for it. This article pretty much sums up my impression of the divide: http://brenocon.com/blog/2008/12/statistics-vs-machine-learning-fight/

For reference: http://brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-and-most-el…

Here is a nice 4-year old still active discussion on pros and cons of different data analysis technologies: http://brenocon.com/blog/2009/02/comparison-of-data-analysis-packages-…

Well, it shouldn't be faster than "Read 1,000,000 bytes sequentially from memory" (741ns) which in turn shouldn't be faster than "Read 1,000,000 bytes sequentially from disk" (359 us). That said, all those numbers feel a bit off by 1.5-2 orders of magnitude -- that disk read speed translates to about 3 GB/s which is well outside the range of what HDDs can achieve. https://brenocon.com/dean_perf.html indicates the original set of numbers were more like 10us, 250us, and 30ms. And it links to https://github.com/colin-scott/interactive_latencies which seems like it extrapolates progress from 14 years ago: // NIC bandwidth doubles every 2 years // [source: http://ampcamp.berkeley.edu/wp-content/uploads/2012/06/Ion-stoica-amp-… // TODO: should really be a step function // 1Gb/s = 125MB/s = 125*10^6 B/s in 2003 which means that in 2026 we'll have seen 11 doublings since gigabit speeds in 2003, so we'll all have > terabit speeds available to us.

Statsmodels has a nice implementation of logistic regression more suited for statistical analysis. It has no default regularisation, nice labelled coefficients table, and model evaluation metrics. https://www.statsmodels.org/dev/generated/statsmodels.discrete.discret… This reflects the difference in intended uses. I view Sklearn as a collection of tools to help forecast a target, whilst statsmodels is collection of tools to examine a model you have of the data-generating process. If you want to do the former, sensible defaults are helpful - for example random forest has many tune-able parameters, and I am glad maintainers haven't shrugged and left all unspecified, because I know I can quickly fit a reasonable first-cut model. For latter, you are better off using statsmodels. In a way this is part of the culture difference between statistics and machine learning. http://brenocon.com/blog/2008/12/statistics-vs-machine-learning-fight/ One common pain point with sklearn is that it uses numpy arrays as representation of data series, whilst a lot of the ecosystem create data cleaning and prepare their data as pandas dataframes.

My gut-ballpark was going to be around an order of magnitude, not two. Here's are two naive comparisons (granted, from the late aughts and not a direct comparison but a more general one) that show ~7-8x[0,1]. The overhead of PyStringObject is not trivial[2] (though the implementation details likely have changed between Py2.x and Python 3). For things like building accumulators a set of data/log parsing rather than data munging (hits per hour or enumerative tasks), I'd imagine (g|n)?awk might hit your 100x since you'd just grab the fd and traverse being IO bound. I'm not sure how awk does it, but if it's just saving an accumulator value (or 10) in a register. Assuming x64-64 treats, say, an r3 fetch analogously to a fetch to ecx (err..rcx now I guess), rather than having to keep a full object in L1 cache, awk has a huge advantage. --- N.b., if you're benching tasks like this, don't use 'time' and STDOUT and think you're getting real performance numbers. Your bottleneck (terminals can only render $x lines a minute, so the kernel call to write(STDOUT, ....) will be where you choke, not at the language. Also disk fragmentation would be another issue. Put the both the test file and the output file on a RAMdisk.) Cache flush with Something like sync; echo 3 > /proc/sys/vm/drop_caches (on Linux, I forget the BSD way of doing it) then `time benchmark.py /mnt/ramdisk1/file > /mnt/ramdisk2' over multiple runs, under various loads, with different data sets, etc Another interesting…

Interesting article! However, Deep Blue, Watson, and AlphaGo are very different from one another. I don't think anyone deemed beating humans at chess or jeopardy impossible at the time Deep Blue and Watson were built. On AlphaGo the point about generalizing AI is valid, yet I think the author doesn't fully appreciate the novelty of approach described in the AlphaGo paper. Their work advances the field, has more general utility than Deep Blue's chess or Watson's jeopardy programs. AlphaGo paper specifically represents an advance in machine learning algorithms for games in general. As I understand, Watson's new NLP algorithm is PRISMATIC [1]. PRISMATIC is a rule-based NLP system and AlphaGo is more statistical inference/neural networks. Even if AlphaGo's 'policy-network- value-network' framework is not too generalizable, the philosophical implication is that we can build AIs that can mimic 'human intuition'. Jeopardy and chess have lesser components of 'human intuition' than Go. They are apples and oranges. So, I wonder if the author errs in bringing Deep Blue and Watson-Jeopardy into the picture. In my opinion while Watson-Jeopardy and Deep Blue were 'over-fitting' for Jeopardy and chess respectively, AlphaGo algorithms are more general and over-fits for the larger category of 'games'. [1] http://brenocon.com/watson_special_issue/05%20automatic%20knowledge%20…

Here's a few things that really helped me out: http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example… http://brenocon.com/blog/2013/10/tanh-is-a-rescaled-logistic-sigmoid-f… Sort of a high-level overview (some of this is particular to neural nets as they're implemented in JMP, a product made by SAS, I don't know how well they generalize): • Neural nets consist of an input layer, an output layer, and some number of hidden layers between the input and output layers • The hidden layers consist of a number of nodes, which can be thought of as “on-off” switches (called a “step function” if we’re using the proper term), although in practice they’re represented by a smooth sigmoid curve with an upper and lower bound that represent on and off. In JMP, this takes the form of a hyperbolic tangent function, which is bound between -1 and 1. • Each node has a value that’s called the bias, which if you’re thinking of the nodes as an on-off switch, is the threshold a linear combination of the inputs needs to reach to turn the node on. In the sigmoid curves output by JMP, the bias represents the point at which the hyperbolic tangent function returns 0, given a linear combination of the inputs. • A “deep learning” model is one that has more than one hidden layer; the lowest hidden layer serves as the input layer to the next hidden layer, and so on. • The output layer takes the nodes of the highest hidden layer and produces an output. When dealing with probability,…

First, I must say that I like the idea of the language. It has ADTs (case classes) and typeclasses (traits), a bit of type inference, can run on Android and isn't Java. It's just the baroque syntax that bothers me. It starts from this [1]. Then there's case classes, _, optional everything (forgetting = in a function def can make it mean something else), the currying syntax [2] (sometimes it needs an arbitrary _ to break ambiguity), the imports (especially renaming), arbitrary rules on whether construction needs "new" or not, etc. The fact that [3] is possible even though Scala is not an almost-no-syntax language (like Lisps and Haskell) concerns me. And it even has macros, but all of its very many language features aren't implemented as macros :( 1. http://en.wikipedia.org/wiki/Scala_(programming_language)#Syntactic_fl… 2. http://brenocon.com/scalacheat/ 3. http://blog.fogus.me/2009/03/26/baysick-a-scala-dsl-implementing-basic…