Tidyverse Blog

We’ve had a lot of scientist use R and the “tidyverse” collection of packages. Ggplot2 is fantastic for graphing. I thing a lot of them used “rstudio” to browse the data. https://www.tidyverse.org/

There's nothing about RStudio that encourages big single files or writing huge unstructured scripts. RStudio is a pretty good IDE, and R is a highly expressive functional-first [0] language. R was heavily influenced by Scheme, and has its own powerful metaprogramming [1] system - which is used to great effect in Tidyverse[2] libraries to make APIs that are nicer and convenient than anything reasonably practical in Python. The problem with a lot of end-user R code is that it is written by statisticians, not programmers. They'd write the same garbage and huge scripts in Python (trust me, I know). [0] http://adv-r.had.co.nz/Functional-programming.html [1] https://adv-r.hadley.nz/metaprogramming.html [2] https://www.tidyverse.org/

R is my go-to for this, and one I recommend often. It makes a nice calculator language (with only a little weird syntax), has a very rich standard library of functions, and is reasonably fast. The great thing about R is that it scales well with task difficulty. It works well for a couple numbers. Then you can use data frames, and the Tidyverse (https://www.tidyverse.org/) packages to do a lot of data analysis (especially dplyr, which is extremely powerful). Finally, if you want to look at your data, the built-in plotting capabilities are solid, and ggplot2 is both extremely powerful and reasonably easy to use.

I highly recommend anybody getting into R to skip the base language (which indeed is ancient and full of gotchas) and go straight for the Tidyverse[1]. You can always go back in and learn how to do things the old way later. Over the last decade, the R community has largely standardized around tools like dplyr, ggplot, tibble, purrr, and so on that make doing data science work way easier to reason about. Much more ergonomic. At my company we switched from using Python to using R for most analytical data science work because the Tidyverse tools make it so much easier to avoid bugs and weird join issues than you get in a more imperative programming environment. [1] https://www.tidyverse.org/

The joke about R is that it is great because it was written by statisticians for statisticians and that R is terrible because it was written by statisticians for statisticians. The two best reasons to use R, IMO, are that many statisticians write up their new methods in R, so it is a window into current statistical research and practice [0], and that R is home to ggplot and the rest of the tidyverse (or Hadley-verse), a systematic approach to common data analysis tasks. [0] https://cran.r-project.org/web/packages/available_packages_by_name.htm… [1] https://www.tidyverse.org/

I loved the idea of Julia but R and specifically the tiddyverse https://www.tidyverse.org/ Just makes everything else seem not as elegant to my humble eyes.

I'd second the R recommendation. It's a particularly exciting time to use R, with the proliferation of package collections like tidyverse (https://www.tidyverse.org/) and the development and release of free texts like R for Data Science (https://r4ds.had.co.nz/).

True, the default semantics of R's data structures are somewhat arcane (of course as they're based on S [1] from the 70's). And the current support for e.g. 64bit integers leaves something to be desired. But behind the scenes, R is just a lisp with some data structures that are adapted to statistics and data science. All base data structures are by default immutable. And e.g. the vector type is extremely performant as it's just a thinly wrapped C Array. In Python you need to reach for Numpy for anything similar, and you do feel some pain when converting between native python types and Numpy types for various functions which support one or the other. The data frame is immensely powerful. And has excellent performance characteristics as it's built upon vectors. A list of objects, like you'd make in python is just a lot slower and more unwieldy to deal with. And much harder to make generalizable functions upon. Hadley Wickham's Tidyverse[2] is exactly an attempt to hide away the arcane details and create a modern, coherent and consistent language on top of R, keeping the power of all the great statistics R libraries. The fact that R behind the scenes is a Lisp, with support for macros, makes this possible. For doing data-transformations and statistics, I can't think of anything currently as powerful as CRAN + Tidyverse. [1] https://en.wikipedia.org/wiki/S_(programming_language) [2] https://www.tidyverse.org/

I'm a scientist (PhD student in microbiolgy) that works with lots of data. My data is on the order of hundreds of gigabytes (genome collections and other sequencing data) or megabytes (flat files). I use the `tidyverse` from R[0] for everything people use `pandas` for. I think the syntax is soooo much more pleasant to use. It's declarative and because of pipes and "quosures" is highly readable. Combined with the power of `broom`,fitting simple models to the data and working with the results is really nice. Add to that that `ggplot` (+ any sane styling defaults like `cowplot`) is the fastest way to iterate on data visualizations that I've ever found. "R for Data Science" [1] is great free resource for getting started. Snakemake [2] is a pipeline tool that submits steps of the pipeline to a cluster and handles waiting for steps to finish before submitting dependent steps. As a result, my pipelines have very little boilerplate, they are self documented, and the cluster is abstracted away so the same pipeline can work on a cluster or a laptop. [0] https://www.tidyverse.org/ [1] http://r4ds.had.co.nz/ [2] http://snakemake.readthedocs.io/en/stable/

R is the PHP of data science. It is productive, it has a large ecosystem, lots of functionality, but it grew fast and organically and not in well planned manner, making it not consistent and a bit messy to work with. If you have to use R, use the tidyverse. https://www.tidyverse.org/ I like R and use it often as it find it more concise to work with than Python for simple statistical purposes. I forced myself to use R instead of spreadsheets and don't regret it. This is one the reasons why (thanks, Zed Shaw) https://web.archive.org/web/20110702162929/https://zedshaw.com/essays/…

What do you think of webR?: https://www.tidyverse.org/blog/2023/03/webr-0-1-0/ Some great showcases: https://github.com/coatless-r-n-d/webR-quarto-demos/issues

... and the following is what I mean. I think this is as close to clean and understandable as it can get, for most use cases (variables in functions, including creating new variables): https://www.tidyverse.org/blog/2020/02/glue-strings-and-tidy-eval/

Tidyverse is so overrated that I don't even know where to start... sure dplyr is nice to use if you're working interactively and ggplot2 has good first principles as the basis of its design, but that's about as far as the tidyverse niceness goes. The apis, the documentation, the inconsistencies (on a language level, between tidy packages and within each package), the problems with backwards compatibility, the evaluation (standard vs non-standard vs tidyeval) - it's a big mess. If you're writing code that's deployed in any way, it's best to avoid the tidyverse as much as possible. This is also acknowledged to some extent by the main developer - https://www.tidyverse.org/articles/2018/06/tidyverse-not-for-packages/

Data frames in R give you the relational model with a Turing complete language. I think of it as a better SQL, at least for analytics (as opposed to transaction processing). Besides its data structures, R is surprisingly similar to JavaScript -- it's has Scheme-like semantics but C-like syntax. For certain problems, it's a pleasure to program in. You don't have to go through "objects" to get to your tables; you just have tables! The examples here might give you a flavor: https://cran.r-project.org/web/packages/dplyr/vignettes/dplyr.html R itself has a lot of flaws, but the tidyverse [1] is collection of packages that provide a more consistent interface. FWIW I agree that more mainstream languages should have tables natively. It is an odd omission. Somehow they got coupled to storage, but they're also useful for computation. [1] https://www.tidyverse.org/

And base R has had a pipe for a couple years now, although there are some differences between base R's |> and tidyverse's %>%: https://www.tidyverse.org/blog/2023/04/base-vs-magrittr-pipe/

Trying to find a good reference for you. There’s this from the author of dplyr: https://news.ycombinator.com/item?id=30067406 And here in the docs: https://dplyr.tidyverse.org/reference/dplyr_by.html And maybe also: https://www.tidyverse.org/blog/2023/02/dplyr-1-1-0-per-operation-group… I guess part of it is that there’s some ‘non-locality’ in the pipeline where the grouping could be relatively distant from the operation acting on the grouped data. Similarly, you get to worry about eg grouping data that is already grouped. I quite like the prql solution which is to have a ‘structured grouping’ where you have to delimit the pipeline that operates on grouped data, but maybe it can still lead to bad edits for complex queries.

There are few differences in performance due between the base pipe and magrittr due to the work we did in 2.0.0: https://www.tidyverse.org/blog/2020/11/magrittr-2-0-is-here/. This release also cleaned up some edge cases so that the semantics are as similar as possible too, making it easy for folks to switch between the two.

Under the hood, the tidyverse is comprised of many packages, some very large and sprawling, like ggplot, and others that do one single thing (readr). Here’s a list: https://www.tidyverse.org/packages/ Individual packages can be used in isolation, if that’s what you need. However, they complement each other nicely, in that they share data representations and similar conventions while handling different facets of data analysis. Thus, people often import the whole thing in one shot as the tidyverse and go from there. For example, you might read the data in with readr, clean it up with tidyr or dplyr, analyze it with tidymodels, and plot the results with ggplot. The python equivalent would be a hypothetical meta-package that tightly integrates pandas, seaborn, and sklearn.

Base R only? tapply? Ick, no thanks. New R programmers should learn tidyverse https://www.tidyverse.org/learn/, not just base R. There are a lot of R programmers (including the author of the OP, apparently https://github.com/matloff/TidyverseSkeptic) who are used to the old way of doing things and allergic to tidyverse. But the base-R only, anti-Tidyverse attitude is going the way of COBOL. I have worked full time in R for many years and it is no contest. I agree with the author that one must learn vectors and subscripts, but that is not an excuse to shove dplyr under the rug. It is an easier way to do lots of things.

The R "userland" pipe magrittr has been altered to make it more compatible with the proposed base pipe, as well: https://www.tidyverse.org/blog/2020/11/magrittr-2-0-is-here/

There is a new operator, {{ or “curly curly”, aimed at dealing with this. It’s part of the rlang (0.4) package and looks snazzy / easy to grasp. Check it out: https://www.tidyverse.org/articles/2019/06/rlang-0-4-0/