Chris Stucchio

I love this, for the reasons laid out in this article (which I always back to): https://www.chrisstucchio.com/blog/2014/why_xkcd_style_graphs_are_impo…

If by some flaws you mean the same logic proves barometric pressure does not exist. https://www.chrisstucchio.com/blog/2019/p_a_statistical_myth.html > Anyone following Nassim Taleb's dissembling on IQ lately has likely stumbled across an argument, originally created by Cosma Shalizi, which purports to show that the psychometric g is a statistical myth. But I realized that based on this argument, not only is psychometrics a deeply flawed science, but so is thermodynamics!

There is no design of AI that can be simultaneously utilitarian, procedurally fair, and representatively fair. I'm going to repeat this again, since many people struggle with this. It is _literally impossible_ to achieve the best utilitarian outcome, the most procedurally fair outcome, and a representatively fair outcome. Anyone designing algorithms will have to make tradeoffs along this frontier. For a far better argument for the above than I can make in a Hacker News thread, please see these slides from Chris Stucchio: https://www.chrisstucchio.com/pubs/slides/crunchconf_2018/slides.pdf

The COMPAS analysis they refer to is exactly one of those cases where there is no bias whatsoever and they're trying to force the algorithm to conform to PC instead of reality. What the ProPublica analysis showed was that a black person and a white person with the same risk score have the same probability of recidivism. There is no racial bias. Black people have higher rates of recidivism, which means that they will also have a higher rate of false positives. This is mathematically unavoidable. I recommend creating a toy example in excel and playing around with some numbers to get an intuition for it. Also check out this analysis: https://www.chrisstucchio.com/blog/2016/propublica_is_lying.html

Whenever I talk to people about downsides of A/B testing, I always try to talk to them about this slide: < https://www.chrisstucchio.com/pubs/slides/gilt_bayesian_ab_2015/slides… > This one visual sums up the most ubiquitous naive use of A/B testing, which I've seen practiced everywhere I've worked. The whole slide deck is very worthwhile.

No, he's saying HFT are great and should be kept because they are a net gain for society. They lower spreads and take margins away from the huge players. So much so, that the big players are lobbying for taxes like this to make HFT less effective so they can make more money for themselves. HFT is just another example of software eating the world and doing a job that used to be done by humans more efficiently. This is a good introduction to the mechanics of what HFT is, written by... yummyfajitas on HN I think? https://www.chrisstucchio.com/blog/2012/hft_apology.html

This is simply not true. Most algorithms can and will correct for biases in their inputs. See this (somewhat technical) article where I go into explicit (simulations in numpy) levels of detail: https://www.chrisstucchio.com/blog/2016/alien_intelligences_and_discri… The best analogy I've come up with for the non-technical is that algorithms are like humans trying to draw inferences on octopus society. Some octopi might have bias against some other octopi, but it's the height of octopusthromorphism to to expect a human to reproduce that bias.

Disallowing the use of race as a signal in machine learning That sounds like a great idea, but it isn't as simple as it appears. Most systems won't have "race" directly encoded as a feature, but that is insufficient. See slide 22 from [1], further in depth discussion discussion from [2] or just this feature (which many systems could automatically discover): Feature6578 = Loc=EastOakland && Income<10k I don't know what the solution to this is, but it's a pretty hard problem, and even the best intention are insufficient. Delip proposes in [1] to introduce a "fairness constraint" where the probability of a favourable outcome in the majority class is the same as in a protected class. This sounds sensible, but make optimising systems much harder. [1] http://deliprao.com/archives/129 [2] https://www.chrisstucchio.com/blog/2016/alien_intelligences_and_discri…

If anyone hasn't already read Chris Stucchio's explanation of market-making/HFT, this is well worth a read. https://www.chrisstucchio.com/blog/2012/hft_apology.html https://www.chrisstucchio.com/blog/2012/hft_apology2.html https://www.chrisstucchio.com/blog/2014/how_to_not_get_ripped_off_by_h… A basic summary is that market-makers add liquidity to the market and profit from their bid-ask spread, increasing the execution-speed and depth of the market. Many of the 'predatory' pricing strategies attributed to them by Lewis and others appear to be impossible when you try and write down the pseudo-code + order book that corresponds to the allegation.

Here are 2 good articles that follow up on the arguments presented by VWO in that article. From the first link below: "They do make a compelling case that A/B testing is superior to one particular not very good bandit algorithm, because that particular algorithm does not take into account statistical significance. However, there are bandit algorithms that account for statistical significance." * https://www.chrisstucchio.com/blog/2012/bandit_algorithms_vs_ab.html * https://www.chrisstucchio.com/blog/2015/dont_use_bandits.html

I love your posts, and completely agree that Bayesian Bandits are not replacements for A/B tests. To be fair though, the realworld issues like nonstationarity and delayed feedback are also concerns for A/B tests (which you also bring up in your great post), and you can tweak the bayesian bandits to handle these cases decently. How does counting sessions instead of visitors avoid delayed feedback? I read your post [0] but dont remember anything about that. Is it just that they say that after a session is completed (which is somewhat nebulous to measure in many cases), then you have all the data you need from the visit? [0] https://www.chrisstucchio.com/blog/2015/no_free_samples.html

Akka is sold as a concurrency abstraction that works great ay any level: one core, multicore, multi-host. After using it for years, I can say it does work, but only with a high complexity overhead. Failure conditions, retry logic, serialization and delivery guarantrees are a concern of multi-host parallelism, but not really multicore (same process) parallelism. I'm not sure I've ever seen an abstraction over memory-separated concurrency that doesn't add a lot of complexity. It may be that such a difference is fundamental and isn't meant to be abstracted over. In any case, unless you are sure that you want to design for distributed parallelism, I recommend conventional thread pools, etc. Also, https://www.chrisstucchio.com/blog/2013/actors_vs_futures.html

That seems partly a reference to this classic article <https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html> "Your data isn't that big". But in this case, it does seem incredibly advantageous to be able to scale up to any number of parallel searches, and to be able to search arbitrary new genomes.

I'm not sure about the macro-level benefits of this kind of high speed trading. Yes, there are winners and there will be losers. The common-folk won't be able to win consistently with high-speed trading strategies. But, FOREX has been like this for years already. This made me think of the great series here: http://www.chrisstucchio.com/blog/2012/hft_apology.html

If what you are looking for is "narrative" non-fiction, Flash Boys and Dark Pools sort of own the corner. Of the two, Dark Pools is much better. In general, it is hard to tell a good narrative around HFT because it is so technical. Trading & Exchanges by Harris is dated but will give you an introduction into market microstructure, keep in mind it was written before RegNMS. If you want a good set of blog posts, Chris Stucchio (yummyfajitas) has an intro series that is great on his blog: http://www.chrisstucchio.com/blog/2012/hft_apology.html

High frequency trading is a different game, though: quoting from an excellent article [1]: "Most HFTs run a market making strategy. What this means is they play both sides of the table - they take no position on whether a stock will go up or down. Instead, they try to offer securities both to buy and sell. If you want to buy, they will sell to you at $20.10. If you want to sell, they’ll buy from you at $20. As long as their buys and sells match don’t get too out of whack, the HFT will collect $0.10 = $20.10 - 20.00." [1] http://www.chrisstucchio.com/blog/2012/hft_apology.html

I'm majoring in CS and Finance too and if you haven't checked these HFT articles out yet, they are 100% worth the read and are very interesting: http://www.chrisstucchio.com/blog/2012/hft_apology.html http://www.chrisstucchio.com/blog/2012/hft_apology2.html

I don't know very much about HFT, and these books are old, so take this with a cow lick of salt, but I've seen these mentioned on a blog I found here on HN (by 'yummyfajitas), and I found them illuminating: Algorithmic Trading & DMA by Barry Johnson Trading and Exchanges: Market Microstructure for Practitioners by Larry Harris You might also be interested in 'yummyfajitas blog: https://www.chrisstucchio.com/blog/2012/hft_apology.html If people have up to date replacements for these books, I'd be super interested. If you keep asking around, people are gunnuh recommend Reminiscences of a Stock Operator, and it's an enjoyable book but keep in mind that it's basically the fictionalized biography of a problem gambler who happened to operate in the stock market, he traded recklessly and died destitute (by suicide - when you reread it, you start to notice how he never mentions enjoying anything, or talks about the people in his life in more than passing). Don't take it too seriously.

I seem to remember a blog post one day about someone interviewing for several "Big Data" positions - where the data turned out to be some Excel sheets. Or fit on a thumb drive. The hiring companies that thought they had "big data" really didn't, was the point. The blog post was widely shared around my office at the time and I see it as the origin of this idea. Edit: I think it was https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html, HN discussion https://news.ycombinator.com/item?id=6398650 But the idea wasn't that data was big as soon as it didn't fit in Excel anymore - Excel was a reductio ad absurdum. If it still fits in Excel, it's laughably far away from being "big". That was the idea. Big data was about the problems you get when you need to join data together that you can't fit well in one database server, not even with the amounts of memory and disk space you can get these days. It was Google type problems, the stuff that map-reduce needed to be invented for. The kind of problem that most companies just don't have. So the term mostly disappeared from job descriptions, "data science" became popular instead, and people keep using Excel because it is great. (that blog post is from 2013 so it can't be the source of that quote from 2012. But please don't define big data in terms of Excel in a 2021 MSc thesis)

This is very misleading reporting. First: All studies so far show a pretty consistent ~10% work disincentive. This is what all the detractors say when they say it disincentivizes work. So how about this one? From actually reading the study's conclusion: > Slightly less than one-fifth were employed before but unemployed during the pilot (17%) So even worse than what we've seen so far. 17% dropping out of the labor market when its a short-term study is huge. For the ~10% figure, Chris Stucchio has a fairly succinct roundup of the work disincentive of other studies so far: https://www.chrisstucchio.com/blog/2019/basic_income_reduces_employmen… ~~~ Personal opinion: If you consider multi-generational entrenchments of poverty as its own problem, worth serious merit, then the work disincentive could be a disaster. In UBI long run, the children of parents who have never worked are probably going to be at a large disadvantage. I think its already a problem today for children of SSI recipient parents (even compared to children of SSDI parents), but its not easy to prove.

They can actually work together. Thompson Sampling / Multi-armed bandit algorithms are traffic allocation strategies to use the least amount of traffic to find the optimal variant. So say you have version A/B/../N variants of a website, you can split the traffic equally or use a a bandit algorithm and find the best performing variant of it. But that only helps you test N variants. If you want to test for 4 titles and 3 color buttons, 3 layouts and 5 images you already have 4x3x3x5=180 different variations and you can't test them all. Evolutionary algorithms can help you search through a much bigger search space: First you try 10 variations, and allocate traffic equally or through a bandit algorithm. Then you combine the top performing candidates multiple times to create a new generation and start over again. Evolution helps you find which candidates to try in a large search space and then a bandits algorithm can help you allocate traffic optimally. You have to be careful with A/B testing with Bandit algorithms though. If the conversion rate changes over time or if you have visitors that don't always convert instantly you have to take that into account: https://www.chrisstucchio.com/blog/2015/dont_use_bandits.html

Humans are a thousand times worse than algorithms. Unattractive defendants get prison sentences twice as long as attractive ones. Judges give significantly harsher sentences just before lunch, when they are hungry. And of course the classic biases against race and gender and other protected classes. I bet there are many other biases that we haven't even discovered yet. Some studies have shown that job interviews are worse than just looking at resumes. Humans should never ever be in charge of sentencing decisions, or anything else you want to be "fair". There's nothing remotely fair about human judgement. One particularly weird bias humans have, is we strongly distrust machines. Psychologists have studied this and named it "Algorithm Aversion": http://opim.wharton.upenn.edu/risk/library/WPAF201410-AlgorthimAversio… Even when we know machines do better than humans, we are much more distrustful of them. We can watch humans make mistakes and an algorithm make many fewer mistakes, and still end up trusting the human far more. And it's weird, because in almost every domain that's been studied, algorithms do vastly better than humans. For any statistical problem where there are a few features available to predict the outcome, even simple linear regression typically beats human experts. We suck at probabilistic and statistical problems. Often the predictions of experts are barely better than chance. And this has been known for a long time. The…

You have been arguing against basic income for years now, and with some really good points[1], but I think you consistently miss the main attraction of the idea to the HN crowd. Many here believe that if people weren't forced to work for income, they would be doing things that are more valuable but also more difficult to monetize (at least initially but perhaps generally), like open source development. There is an underlying belief that vast creative talents are untapped or wasted because of the need to work for money. It's the intellectual's dream lifestyle. One that used to be possible for writers and academics. Other reasons are largely an afterthought. That's why basic job for example is not a convincing counterargument despite the numbers. This is not really about helping the poor, that would be an auxiliary benefit at best. [1] https://www.chrisstucchio.com/blog/2013/basic_income_vs_basic_job.html

I tend to agree that the value of a company doesn't actually change substantially from one second to the next. But to focus on that is to ignore the value of having liquidity available. For an example of what can happen when there isn't enough liquidity, take a look at the graph of RSP on the morning of August 24th. For a more detailed description of the mechanics and motivation behind market making, I recommend "A High Frequency Trader's Apology": https://www.chrisstucchio.com/blog/2012/hft_apology.html If traders, be they high-frequency or otherwise, can't make money by making liquidity available, they won't do it. Less liquidity means wider spreads, which means higher costs especially for smaller transactions (i.e. individual investors). You could reasonably argue that HFT has caused expenses to increase for large investors who need to buy or sell large volumes of shares. But to the extent that is true, it is like saying that large investors used to get on average an unreasonably good deal at the expense of their counter parties. As in any free market, an especially good price for one party is by definition an especially bad price for the other party.

When you first wrote about how hadoop is a waste of time if you don't have multi-TB worth of data (http://www.chrisstucchio.com/blog/2013/hadoop_hatred.html ), I thought that was classic linkbait. And then I actually started seeing real benefits of not doing map-reduce for smaller datasets ( small = 1GB-1TB) & just sticking to plain old scala ( poso as opposed to pojo :) Similarly, this article seems linkbait on the surface but makes a lot of sense if you do anything performance intensive. I recently tried implementing a multi layer neural net in Scala - I eventually ended up rewriting your mappers as while loops & introduce some mutables, because at some point, all this FP prettiness is nice but not very performant. It looks good as textbook code for toy examples, but takes too long to execute. Am still a huge fan of FP, but nowadays I don't mind the occasional side effect & sprinkling some vars around tight loops. Its just much faster, & sometimes that is important too.

That's rather simplistic, not to mention completely wrong. US markets are highly visible (you just saw a free visualization of HFT activity since 2007), very tightly regulated and well controlled. Things go wrong, as they do in all complex systems, but rarely do they go wrong to the extent that ordinary investors are harmed. Read this, before jerking your knee any further. Think of it as turning on the bedroom light, showing the monster in the corner to be nothing more than a pile of clothes: http://www.chrisstucchio.com/blog/2012/hft_apology.html http://www.chrisstucchio.com/blog/2012/hft_apology2.html Financial markets have been moving towards zero latency trading since their inception. First, men gathered under trees or in coffee houses, within shouting distance of each other. They fought over the best spots. Then, traders became some of the earliest adopters of telegraph technology, running private lines from remote cities to the exchanges. Ticker tapes appeared in distant offices. Computerized price dissemination followed, together with electronic trading connections to the exchanges themselves. All in the name of getting information and acting on it before the next guy. It's the reason you, as an individual, can trade global markets cheaply and instantly. It helps you take control over your retirement and savings, rather than paying through the nose for some managed scheme where only a privileged few can access the markets. Where is your arbitrary line where…

The problem of HFT is two. First, HFT is very difficult. It requires the talents of extremely intelligent individuals. These individuals are a scarce resource. Now, being as they are near the generation and handling of money, HFT firms are best placed to outbid all others for these people. So R&D and other such lose out on top minds. Additionally, if much of the money of high net people is being invested in funds without finding its way into risky high tech research, this too can be a drag on the economy in the long term. http://sirc.rbi.org.in/downloads/4Cecchetti.pdf The second is a problem of misaligned priorities. It's one thing to suck in all the best minds and it's another to have them all expending a massive amount of effort trying to shave down microseconds so their order goes in first. Chris Stucchio here http://www.chrisstucchio.com/blog/2012/hft_whats_broken.html argues very well that the so called sub-penny rule (abs(changeinprice) >= 0.01) is antiquated and from a time when humans reigned. This rule is a source of friction for the bots and is very likely a key player in the current misallocation of talent. It's removal would make HFTing cheap enough that not so many people would be needed. Some would leave to cure cancer or mine asteroids. The rest could then work on more sophisticated pricing techniques - which is a net boon, maybe those techniques could make prices more stable or be reused for optimal donor matching or something.

Reminds me of this gem from the previous decade: https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html (Don't use Hadoop - your data isn't that big) Which, of course, was discussed on HN back then: https://news.ycombinator.com/item?id=6398650

Also from 2013: https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html

Reminds me of the craze for using Hadoop on data that could easily be handled by basic R, Python, or even Excel https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html In hindsight, the Big Data hype wave was really just a good excuse to bring in a wider set of analytics tools than Excel, Access and inflexible, IT-administered databases.