Win-Vector Blog

I fit a Bass product lifetime model on earlier related StackOverflow data, it looked bad at the time. https://win-vector.com/2025/03/02/best-before-dates-by-bass/

jmount has an explanation elsewhere in this thread linking to https://win-vector.com/2021/12/26/how-to-read-sourav-chatterjees-basic… which does a great job of explaining the intuition, but in a nutshell the normalization factor of 3 comes from the fact that if you select two random points between 0 and 1, the mean distance between will be 1/3 (which is pretty easy to write down and solve, boils down the the fact that a pyramid of height 1 that's 1x1 at the base has volume = 1/3).

R is great mostly for the breath of plugins/implementations contributed by the community... much better than its alternative SAS for an infinitely lower price... ($0 vs ~$10k). Not surprised by this move from Oracle as data analytics/mining gets more popular. For R enthusiasts there is a great blog I follow http://www.win-vector.com/blog/2012/07/modeling-trick-masked-variables…

My favorite definition of math, courtesty of http://www.win-vector.com/blog/2008/08/what-is-mathematics-really/#mor… "Mathematics is the minimal environment to preserve ideas."

I like the asinh scale, I’ve come across it before as “signed pseudolog scale” - your name for it is nice and descriptive, do you have a writeup about it somewhere? https://win-vector.com/2012/03/01/modeling-trick-the-signed-pseudo-log…

> You’ll notice one other small change to the tidyverse message: we now advertise the conflicted package. Not a fan of the nature in which tidyverse promotes itself. They used to be more aggressive - i.e. advertising their books in package load messages [1]. But these new changes still feel like an ad. To me, as an outsider, tidyverse seems to mainly be about "dplyr". The main thing holding all of it together otherwise is just the fact that majority of their functions return a "tibble" instead of a data.frame, forcing you to load all the jungle of the rest of their stuff whenever you use one of their packages. [1]: https://win-vector.com/2019/08/30/it-is-time-for-cran-to-ban-package-a…

Numpy is basically lapack. You'd be hard pressed to replace that with something nearing its performance. For autodiff, I dunno, how about John Mount's explanation? http://www.win-vector.com/blog/2010/06/automatic-differentiation-with-…

I think that linck should be here: http://www.win-vector.com/blog/2014/01/use-standard-deviation-not-mad-…

Here's some commentary on the article: http://www.win-vector.com/blog/2014/12/a-comment-on-preparing-data-for…

This is good as well: http://www.win-vector.com/blog/2012/07/minimal-version-control-lesson-…

While I agree with the sentiment in general, I've came to the conclusion that what I really want is the flexibility of the natural language interface that LLM's provide, and the return of the correct document. No 'reasoning', no summarizing, just better search [0]. The issue with the current generation of models is that they can't reason, they may do very well at pretending to reason, but they can't [1]. Reasoning requires the ability to identify and reuse patterns, and while there has been some advancement in this area [2] with getting models to learn the underlying pattern and rule, it doesn't generalize. This results in models that will happily tell you that a statement is both true and false, and be unable to identify the logical problem with that. Even creating summaries is difficult, and LLM's are more than happy to hallucinate even when summarizing documents, providing incorrect, or entirely made up facts [3]. The general workaround is multiple runs with the same work and averaging the response, but that's a lot of work, and energy. [0] https://win-vector.com/2024/05/21/i-want-flexible-queries-not-rag/ [1] https://medium.com/@konstantine_45825/gpt-4-cant-reason-2eab795e2523 [2] https://arxiv.org/abs/2405.15071 [3] https://community.openai.com/t/gpt-4o-hallucinating-at-temp-0-unusable…

Thanks! Yes, the concerns you mentioned would also apply to PCA. What we've actually done to help alleviate this is a union of components from y-aware[1] and normal PCA to capture variables that are correlated to both the dependent variables and (hopefully) most of the treatment variables. This seems similar to the double selection approach you mention - the difference being that since we are trying to run this at scale for 1000s of treatment variables, running a feature selection with each of the 1000 treatment variables as the dependent variable isn't super feasible, so the normal PCA acts as proxy for this part of the double selection. Regardless, we're never going to completely remove omitted variable bias, as we're never going to capture 100% of relevant variables. One way we monitor our model's bias is by looking at the error distribution between users in the treatment vs control. If these aren't similar, there's too much bias in our estimate of the treatment effect, so we wouldn't want to serve an estimate of the treatment effect for this variable to our customers. The current product is in beta and we're working with some of our current customers to try to re-create our results with A/B tests. I'm hoping that by our GA release in the fall we'll have some case studies with specific examples! [1] http://www.win-vector.com/blog/2016/05/pcr_part2_yaware/

These are a few in my bookmarks: Quick-R http://www.statmethods.net/index.html R for Programmers http://en.wikibooks.org/wiki/R_Programming SurviveR http://www.win-vector.com/blog/2009/09/survive-r/ Official intro http://cran.r-project.org/doc/manuals/R-intro.html Might depend on your definition of accessible LOL Stack Overflow also very helpful - http://stackoverflow.com/questions/tagged/r Rseek http://www.rseek.org/ R-Help http://r.789695.n4.nabble.com/R-help-f789696.html