Jake VanderPlas

Jake and I worked together around this time on similar topics and all I can say is: he is a really great mentor and superhuman generator of incredible work. FFT is related to Lomb Scargle (least squares fitting of sinusoids when you have non-equispaced time series data), he has an excellent paper on Lomb Scargle as well that I recommend. Here’s a good associated blog post: https://jakevdp.github.io/blog/2017/03/30/practical-lomb-scargle/

Slightly off topic, but whenever I see talk about perceptually-uniform color, I'm reminded of this classic post about graphing library colormaps: http://jakevdp.github.io/blog/2014/10/16/how-bad-is-your-colormap/ (which also links to https://matplotlib.org/2.0.2/users/colormaps.html)

Worth mentioning is the The Waiting Time Paradox: "When waiting for a bus that comes on average every 10 minutes, your average waiting time will be 10 minutes." And worse "when the average span between arrivals is N minutes, the average span experienced by riders is 2N minutes." These realizations about Poisson distributions have a lot of real-world implications such as packet traffic, call center calls, etc. - https://jakevdp.github.io/blog/2018/09/13/waiting-time-paradox/

Jake Vanderplas did something similar many years ago in python [1]. I miss his blog posts (must be busy doing something great). [1] https://jakevdp.github.io/blog/2012/10/07/xkcd-style-plots-in-matplotl…

Matplotlib has XKCDify which is similar[0]. It's great for rough or estimated graphs that are not to be read accurately. [0]: https://jakevdp.github.io/blog/2012/10/07/xkcd-style-plots-in-matplotl…

Follow up in 2013. XKCD officially merged in Matplotlib. http://jakevdp.github.io/blog/2013/07/10/XKCD-plots-in-matplotlib/

> so then why does it matter? I recommend reading this[1] essay which explains exactly the difference between the two methods. (I'm linking to part 3 because it specifically answers your questions, but the entire series is worth reading) Why does it matter? Frequentist and Bayesian methods offer different interpretations of what your result means. ... speaking broadly, frequentists consider model parameters to be fixed and data to be random, while Bayesians consider model parameters to be random and data to be fixed. This distinction is important, because... > the frequentist and Bayesian methods are the same ...this isn't always true. See [1] for the details. [1] http://jakevdp.github.io/blog/2014/06/12/frequentism-and-bayesianism-3…

And https://news.ycombinator.com/item?id=12025529 therein specified these: The following may also be of interest: - Why Python is Slow: Looking Under the Hood [0] - Fast Python, Slow Python by Alex Gaynor [1] [0] https://jakevdp.github.io/blog/2014/05/09/why-python-is-slow... [1] https://www.youtube.com/watch?v=7eeEf_rAJds

Jake Vanderplas previously wrote an excellent blog post about Python performance and scientific computing: https://jakevdp.github.io/blog/2014/05/09/why-python-is-slow/

And usually you want to avoid invoking the python C API from cython unless you have to use it for some reason. The whole point of the typed memoryviews is that you get direct access to the raw memory buffer and can access it like a numpy array or even like a pointer array. This blog post [0] incrementally shows how to get microoptimizations out of your cython code. In the end, the autogenerated code is usually not too bad too follow since it's a one-to-one transpilation from the cython to C. cython -a helps a ton, or if you can use it, the cythonmagic [1] for the IPython notebook embeds the output of cython -a inline in a notebook, making the iteration process much quicker. [0] https://jakevdp.github.io/blog/2012/08/08/memoryview-benchmarks/ [1] http://ipython.org/ipython-doc/2/config/extensions/cythonmagic.html

The follow-ups are also well worth reading: http://jakevdp.github.io/blog/2014/06/06/frequentism-and-bayesianism-2… http://jakevdp.github.io/blog/2014/06/12/frequentism-and-bayesianism-3… http://jakevdp.github.io/blog/2014/06/14/frequentism-and-bayesianism-4…

Quite a bit has changed in the last three and a half years when it comes to Python packaging. What specifically did you want to highlight here, and why? Anyway, some responses: > there’s one thing that irritates me every single time I revisit the page and that is the misleading recommendation of their own tool pipenv.... PyPA recommends pipenv as the standard tool for dependency management... PyPA still advertises pipenv all over the place and only mentions poetry a couple of times, although poetry seems to be the more mature product. This looks very different now, but I don't think the recommendation was "misleading". The archived page didn't say that the tool was "the standard", just a recommendation. Meanwhile, Poetry lagged behind for years in implementing standards. Pipenv was also focused only on dependency management (which basically == creating venvs + invoking Pip + updating lockfiles), whereas Poetry is designed as a complete workflow tool (== dependency management + build frontend + build backend + package uploader + project metadata management + a bunch of integration + probably stuff I forgot). > You would expect exactly one distribution for Python packages, but here in Python land, we have several ones. Conda came into existence because the scientific Python community needed to solve packaging problems right now (around 2012 - see…

There’s a chain of stuffs happening regarding packaging and the first step in the chain using PyPI limits what one can do to fix it. Conda starts with a different design in how to build a package and also how to bundle it. It specifically handles compiled code much better, because in that non pure Python case, compiler is needed at some point, and that is going to cause problems if you have no control over it. Conda essentially achieve reproducibility, with the necessary compiler considered as part of the dependencies. While in a certain sense of reproducibility can be achieved with PyPI packages, my understanding is that, eg with manylinux, it takes a hope for the best approach and can break in exotic cases, which should never happen with conda packages. > The genesis of Conda came after Guido van Rossum was invited to speak at the inaugural PyData meetup in 2012; in a Q&A on the subject of packaging difficulties, he told us that when it comes to packaging, "it really sounds like your needs are so unusual compared to the larger Python community that you're just better off building your own" From https://jakevdp.github.io/blog/2016/08/25/conda-myths-and-misconceptio…

I... am not sure that I understand what you mean ? That after shuffling you know less about the stack of cards (that used to be at least partially revealed) is a fact that our model must follow or fail at being relevant. It's been hard to find out more information about this, but I did find some : https://www.preposterousuniverse.com/blog/2015/08/11/the-bayesian-seco… Same thing : "And the Bayesian Second Law (BSL) tells us that this lack of knowledge — the amount we would learn on average by being told the exact state of the system, given that we were using the un-updated distribution — is always larger at the end of the experiment than at the beginning (up to corrections because the system may be emitting heat)." Though I also did find another interesting thing : http://jakevdp.github.io/blog/2014/03/11/frequentism-and-bayesianism-a… "in a strict frequentist view, it is meaningless to talk about the probability of the true flux of the star: the true flux is (by definition) a single fixed value, and to talk about a frequency distribution for a fixed value is nonsense" But that's pretty much the case in statistical physics ! A macrostate is actually NOT a state (=microstate), but a probability distribution ! "For Bayesians, probabilities are fundamentally related to our own knowledge about an event. This means, for example, that in a Bayesian view, we can meaningfully talk about the probability that the true flux of a star…

> It is customary when simulating heat flow to use a wacky color palette where red is hot and blue is cold, with all kinds of intermediate colors in between In an otherwise excellent article, this is the only issue I could find. We really need to stop using/recommending/normalizing rainbow color maps (i.e. jet). They actively confuse readers by creating visual artifacts that aren't actually in the data. This article has some great explanations and visuals. https://jakevdp.github.io/blog/2014/10/16/how-bad-is-your-colormap/ The original post uses a rainbow color map to represent temperature-related things because having a diverging color map is often a useful intuition for temperature heat maps. But in that case, we should prefer one of the following diverging color maps listed on the matplotlib site (this list definitely isn't exhaustive, but it is helpful). https://matplotlib.org/stable/_images/sphx_glr_colormaps_004.png More on matplotlib's well-chosen color maps: https://matplotlib.org/stable/tutorials/colors/colormaps.html

When people say cpython is slow they are generally pointing to two things 1) The interpreter is _slow_ 2) You can't achieve real thread based parallelism I think in general people have a high level concept of (1) as static vs. dynamic and interpreted vs compiled is an understandable trade off. CPython as an implementation generally gets type casted as slow for its default dynamism which have understandable negative performance implications[1]. However CPython also gives you lots of ergonomic ways to push your program towards the static/compiled end of the spectrum with things like pandas/numpy/numba/extensions. In general using these correctly put you within the ballpark of _faster_ languages, could you write faster assembly by hand? Sure! Is optimizing in another language worth your time? I don't know. I've never really understood (2) the lack of threading as a problem, multi process parallelism can be accomplished fairly easily if you are CPU bound or projects like uvloop[2] make async tasks fast enough to compete with any web framework out there. Furthermore even though your cpu cycle cost may be levered to a considerable point while operating over hundreds/thousands of servers doing distributed computing right is still hard and developing something like celery/airflow/luigi/dask from scratch is not cheap either. Leaning on CPython's massive ecosystem can massively lower the barrier to entry in a lot of big problems. I think there are plenty of examples of re-writes…

This post highlights an interesting dichotomy in the Python scientific computing community. Everyone knows that PyPy runs faster than CPython for many common tasks [1]. [1] PyPy is on average 7x faster than CPython: http://speed.pypy.org/ But those in the Python community who are serious about scientific computing (or image processing, like my startup) are already using Numpy & Scipy, which provide hand-coded C implementations of most matrix-related operations. Everyone knows that Python for-loops are "slow" [2], and storing a large 2-D matrix as a list-of-list-of-Python-int-object would require a huge amount of memory & indirection. So, Numpy offers an N-dimensional array type, implemented in C: C arrays of densely-packed C primitive types, with for-loops in C to iterate over the matrix elements. Then Scipy builds a lot of Matlab-like functionality as modules on top of this fundamental Numpy array type. [2] Python for-loops are slow: https://jakevdp.github.io/blog/2014/05/09/why-python-is-slow/ So the most expensive operations in a Python number-crunching program are likely already implemented using Numpy & Scipy operations, which run in compiled C (and additionally, often make use of Blas/Atlas/LAPACK/etc, for even greater speedups in sustained number-crunching). But unfortunately, Numpy & PyPy do not naturally work together. Being written in C, Numpy makes substantial use of the CPython C-API -- and in fact, Numpy provides its own C-API [3]! The official Numpy…

A more recent article addressing this essay from the same author: https://jakevdp.github.io/blog/2014/08/22/hacking-academia/ The tl;dr summary from this second article referencing the first: a quick summary is this: scientific research in many disciplines is becoming more and more dependent on the careful analysis of large datasets. This analysis requires a skill-set as broad as it is deep: scientists must be experts not only in their own domain, but in statistics, computing, algorithm building, and software design as well. Many researchers are working hard to attain these skills; the problem is that academia's reward structure is not well-poised to reward the value of this type of work. In short, time spent developing high-quality reusable software tools translates to less time writing and publishing, which under the current system translates to little hope for academic career advancement. I think the HN title is somewhat misleading, the central thesis from the original article is stated as: the skills required to be a successful scientific researcher are increasingly indistinguishable from the skills required to be successful in industry This is probably a stunningly awesome outcome - it means that industry has a place for advanced degree holders who will not find a classic academic position. In the linked essay above, from the same author: the number of PhDs granted each year far exceeds the number of academic…

Looks like it. From https://jakevdp.github.io/PythonDataScienceHandbook/00.00-preface.html… > Copyright 2016

Related: Xkcd-Style Plots in Matplotlib https://jakevdp.github.io/blog/2012/10/07/xkcd-style-plots-in-matplotl… (discussion: https://news.ycombinator.com/item?id=19293129) XKCD Plots in Matplotlib: Going the Whole Way https://jakevdp.github.io/blog/2013/07/10/XKCD-plots-in-matplotlib/ (https://news.ycombinator.com/item?id=11642344)

The gist of it is that you give numpy two arrays, and what operation to apply. Then numpy will figure out what the for loop(s) should look like depending on the shape of the arrays. You can look at various tutorials to see how it works. For example: https://jakevdp.github.io/PythonDataScienceHandbook/02.05-computation-…

It’s because Python object attributes can change any time, as they are accessed dynamically. Nothing can be inlined easily. The object structure is pointer heavy. Here is some old 2014 post: http://jakevdp.github.io/blog/2014/05/09/why-python-is-slow/ As other commenters pointed out, some of these Python features, which are unused 99,99% time, could be sacrified for additional speedup by breaking backwards compatibility.

Another nice post in the same direction by Jake VanderPlas: https://jakevdp.github.io/blog/2017/12/18/simulating-chutes-and-ladder…

Kaggle learn: https://www.kaggle.com/learn "Python Data Science Handbook" (O'Reilly) http://jakevdp.github.io/PythonDataScienceHandbook awesome-python#web-content-extracting: https://github.com/vinta/awesome-python#web-content-extracting https://learnxinyminutes.com/docs/python/ "What is the best way to learn to code from absolute scratch?" https://news.ycombinator.com/item?id=16299880

I mean, you can modify Python code at runtime if you like. This has a good overview of all the nonsense happening under the hood: http://jakevdp.github.io/blog/2014/05/09/why-python-is-slow/

https://jakevdp.github.io/blog/2014/10/16/how-bad-is-your-colormap/ a decade or so later, people are still using jet and it still sucks!

Very good point. Additionally, with insert and delete we might need to apply the rules of broadcasting [0]. For example, if I have a 2-dimensional array and I insert a scalar value by using axis=0, the scalar value needs to be broadcasted to match the column dimension of the array: >>> a = np.array([[1,2],[3,4],[5,6]]) >>> np.insert(a, 1, 11, axis=0) array([[ 1, 2], [11, 11], [ 3, 4], [ 5, 6]]) [0] https://jakevdp.github.io/PythonDataScienceHandbook/02.05-computation-…

These are called perceptually uniform colormaps and are also useful for grayscale printing, as the scale will still be valid. This is now the default in Matplotlib (eg viridis, magma, etc) https://jakevdp.github.io/blog/2014/10/16/how-bad-is-your-colormap/ I have found occasion to use Jet though, it's good for highlighting transitions between close values when the colour scale is large (but not large enough to use a log map).

Have you seen Whirlwind Tour of Python by the same author? - https://jakevdp.github.io/WhirlwindTourOfPython/ > A Whirlwind Tour of Python is a fast-paced introduction to essential features of the Python language, aimed at researchers and developers who are already familiar with programming in another language. The material is particularly designed for those who wish to use Python for data science and/or scientific programming

Tiny nitpick, you'll probably wait longer than five minutes: >In the case of a nominally 10-minute bus line, sometimes the span between arrivals will be longer than 10 minutes, and sometimes shorter, and if you arrive at a random time, you have more opportunities to encounter a longer interval than to encounter a shorter interval. And so it makes sense that the average span of time experienced by riders will be longer than the average span of time between buses, because the longer spans are over-sampled. https://jakevdp.github.io/blog/2018/09/13/waiting-time-paradox/