Wes McKinney

Wes's book is definitely the standard. But I would hold off buying one right now - he's currently working on a (much-needed) second edition, coming out next year (http://wesmckinney.com/). Joel's book is a great resource for preparing for interviews or learning really basic stuff and less of an introduction to the tools.

"But it's surprisingly challenging to publish books on the web in nice, cohesive, tight, easy-to-navigate HTML format." Quarto is one great option for doing that today. Bonus: it can also generate EPUBs and PDFs, all from one set of source files. https://quarto.org/ It's free and open source. https://github.com/quarto-dev/quarto-cli Example books: https://wesmckinney.com/book/ https://jjallaire.github.io/hopr/

Your first paragraph makes perfect sense! I was nodding along. But then your concluding sentence was a bit of a record scratch for me. This all worked as intended! You knew what the project was about - "data frames" - and what might make it attractive to you - the performance claims - and then you went and followed exactly the right path to get the context you needed to understand what's going on with it. It's a big topic that you were able to spin up on to a basic level in 2-3 hours, by pulling on strings starting at this landing page. This is a very successful outcome. I'd also recommend this book: https://wesmckinney.com/book/. It's not about polars, but you'd be able to transfer its ideas to polars easily once you read it.

For use as a file format, where one priority is to compress columnar data as well as possible, the practical difference between Arrow (via Feather?), Parquet, and ORC is still somewhat vague to me. During my last investigation, I got the impression that Arrow worked great as a standard, interoperable, in-memory columnar format, but didn't compress nearly as well as ORC or Parquet due to lack of RLE and other compression schemes (other than dictionary). Is this still the case? Is there a world where Arrow completely supplants Parquet and/or ORC? EDIT: Just found https://wesmckinney.com/blog/arrow-columnar-abadi, which helps answer this question.

10x reflects both experience and expert recommendations. You may recognize the author [1]: > Nowadays, my rule of thumb for pandas is that you should have 5 to 10 times as much RAM as the size of your dataset [1] https://wesmckinney.com/blog/apache-arrow-pandas-internals/

In linux world, conda is the best choice. Here was an interesting take on it, https://wesmckinney.com/blog/conda-forge-centos-moment/

McKinney has been hard at work getting parquet and arrow support in pandas. http://wesmckinney.com/blog/outlook-for-2017/ >Give the kind of people behind Arrow, I would love wrapper that will use Arrow to do all of this...But doesn't matter at the end of the day. pyarrow; pyarrow.parquet (which uses parquet-cpp).

I recently switched from Scala/Spark to the PyData stack and have been loving it. It's been so refreshing to stop having to think about the JVM, SBT, Scala types, and IntelliJ. PyData libraries let you focus on your data. Wes wrote a 10 things I hate about Pandas article back in 2017 that said "pandas rule of thumb: have 5 to 10 times as much RAM as the size of your dataset". Source: https://wesmckinney.com/blog/apache-arrow-pandas-internals/ I'm not sure that rule of thumb is up-to-date anymore. Modern versions of Pandas use PyArrow, another project Wes started, and should be more memory efficient. Wes just started a new company that'll continue building Arrow: https://medium.com/@patterson.joshua/joining-forces-for-an-arrow-nativ… Dask makes it easy to scale Pandas, a la Spark. Each partition in a Dask DataFrame is a Pandas DataFrame, so you can indirectly use Pandas on terabytes of data. Disclosure: I work at Coiled, the Dask company founded by Matt Rocklin. The PyData stack makes machine learning / advanced analytics so much easier than Spark in my opinion. I still love Spark for ETL, but think modern PyData is a better stack for ML (for most companies).

When Wes McKinney wrote about the transition away from python I knew it was real. https://wesmckinney.com/blog/agent-ergonomics/ I still have a special place in my heart for the language and think it’s still got a niche.

Pandas is terrific, yet even its original author has noted inherent shortcomings [1], and there exist alternatives. Polars seems to be the most prominent competitor in the Python DataFrame space, and DuckDB appears to pursue an approach similar to SQLite, but columnar. I am personally working on a solution to a broader problem, which can also be viewed as an alternative to Pandas [2]. [1] https://wesmckinney.com/blog/apache-arrow-pandas-internals/ [2] https://github.com/ronfriedhaber/autark

See also The Mythical Agent-Month https://wesmckinney.com/blog/mythical-agent-month/

The creator of Pandas even bashes it: https://wesmckinney.com/blog/apache-arrow-pandas-internals/

Source code: https://github.com/wesm/msgvault Blog announcement: https://wesmckinney.com/blog/announcing-msgvault/ Tweet where I discovered this exists: https://xcancel.com/wesmckinn/status/2018303622145065455

About time. It always surprises me how long pandas has been able to hold on. Wes McKinney talked about pandas' limitations back in 2017 https://wesmckinney.com/blog/apache-arrow-pandas-internals/

Yes, it's annoying negative feature of many tech products. Of course it's natural to want to speak to your target audience (in this case, data scientists who like Pandas but find it annoyingly slow/inflexible), but it's quite alienating to newbies who might otherwise become your most enthusiastic customers. I am the target audience for Polars and have been meaning to try it for several months, but I keep procrastinating about because I feel residual loyalty to Pandas because Wes McKinney (its creator) took the time to write a helpful book about the most common analytical tools: https://wesmckinney.com/book/

Announcement from Wes: https://wesmckinney.com/blog/joining-posit/ Announcement from Posit: https://posit.co/blog/welcome-wes/

The whole point of Arrow is that it is language agnostic; so of course it is not a "pandas binary format". However, pandas didn't start supporting Arrow "to try to catch up to polars". The creator of pandas, Wes McKinney, came up with the Arrow idea because of the interoperability limitations he came across while working with pandas and the big data ecosystem. Supporting Arrow has been in the works for a long time: https://wesmckinney.com/blog/pandas-and-apache-arrow/ Pandas is an older and larger library with less maneuver capability and millions of users relying on things not being broken.

It's also well-known that pandas is memory-inefficient and Dask would probably do the memory estimation they described for them (after the author dismissed it). They're really just showing they don't really understand these tools that well https://wesmckinney.com/blog/apache-arrow-pandas-internals/

Here is an example: https://wesmckinney.com/blog/avoid-unsigned-integers/

The first edition of this book was a game changer. I still have my hardcopy paperback but haven’t turned to it in years. It’s probably largely out of date but I look at it fondly. Glad to see the material updated with new APIs and Python 3.10. The open access is very nice because it can easily be used for teaching and reference. I would like to see some of Wes’ cautionary tales included into the book. Pandas feels so magical at times that you can uses it to solve anything. That can create problems. Ideas like this should be some appendix to the book. https://wesmckinney.com/blog/apache-arrow-pandas-internals/ Lots of tools in the data space. It feels like choosing JS web front ends. I use pandas quite a lot but less frequently, as my data has gotten larger I turn to spark more. I’m also impressed by the work DuckDB is doing.

I could be misremembering but I seem to recall Wes McKinney saying in some talk that rather than rewrite Pandas to be Arrow-backed it will probably eventually be replaced by newer Arrow-backed libraries some of which might have pandas-like apis. I think the idea was that pandas API is too large and the library too widely used for it to be practical to correct some of the design problems people have mentioned. He'd sketched out a vision for Pandas 2.0 at one point and I think he said that basically that would probably just be a new library. There's a lot of related discussion in this post on his blog. https://wesmckinney.com/blog/apache-arrow-pandas-internals/

Check out the post from author of pandas on pandas, pandas2 and arrow [1]. This might answer your question on what is the significance of arrow, given today pandas are kind of basic ingredient in scientific computing, AI and ML. [1] https://wesmckinney.com/blog/apache-arrow-pandas-internals/

It might be annoying, but it's certainly not bizarre. In the floating point standard (IEEE 754) there are reserved bit values (with hardware support) to represent NaNs. For integers no such thing exists, so you're left with a bunch of different implementation choices all with different tradeoffs. A long time ago NumPy devs choose not to support NaN in integer arrays at all (for maximum performance), and Pandas (starting as a wrapper around numpy arrays) inherited that. For a more technical discussion of the performance implications of one possible implementation, see http://wesmckinney.com/blog/bitmaps-vs-sentinel-values/

"In Pandas, the only change we need to make to our code, compared to a local filesystem, is to replace open_file(..) with h.open_file(..), where h is a file handle to HDFS/HopsFS." HopsFS is a drop-in replacement for HDFS. Here's some more details on native HDFS connectors in Python by Wes McKinney -http://wesmckinney.com/blog/python-hdfs-interfaces/

I wonder how much of this is driven by market need. Pyarrow + Pandas is significantly fast already. http://wesmckinney.com/blog/high-perf-arrow-to-pandas/ Also Pandas 2.0 is going to roll in a lot more utulities for parallel computing. Is there really a need for 50-100x speedups today ?

I went the same route but ended up adopting feather from the makers of Python's Pandas and R's tidyverse, Wes McKinney and Hadley Wickham. https://blog.rstudio.com/2016/03/29/feather/ > SUMMARY: Feather's good performance is a side effect of its design, but the primary goal of the project is to have a common memory layout (Apache Arrow) and metadata (type information) for use in multiple programming languages. http://wesmckinney.com/blog/feather-its-the-metadata/ The approach is data frames on disk storage. > data frames are lists of named, equal-length columns, which can be numeric, boolean, and date-and-time, categorical (_factors), or _string. Every column can have missing values. This fits the needs of all of my CSV usage.

Wes McKinney posted a 10-minute "whirlwind tour of pandas" [0] several years ago that still serves as a pretty good example of what is possible for a practiced pandas user. [0] http://wesmckinney.com/blog/whirlwind-tour-of-pandas-in-10-minutes/

Anyone know how it compares with Apache Arrow (that pandas is moving to) ? http://wesmckinney.com/blog/apache-arrow-pandas-internals/ http://arrow.apache.org/

You may be interested in what Wes McKinney has to say on the topic: http://wesmckinney.com/blog/outlook-for-2017/ (you can read/write using pyarrow)

> large datasets on disk I saw this benchmark a while back comparing Pandas to SQLite in-memory databases. While Pandas did edge out SQLite in several areas, it was by well under an order of magnitude: http://wesmckinney.com/blog/?p=414 Pretty solid performance plus the ability to work with large datasets on disk seemed like a pretty big win to me. I could imagine a set of SQLite extensions (a la spatialite) that could further optimize for various data.frame use cases. As an added bonus, the same libraries would be very portable between different languages--even languages that don't currently have something like dataframes. EDIT: What I don't know about is memory efficiency. Perhaps SQLite isn't, but I'd not bet against?